2024-04-23T21:13:22.314 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T21:13:22.314 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212 --description rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/orig.config.yaml 2024-04-23T21:13:22.347 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T21:13:22.433 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212 branch: main description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit} email: ceph-qa@ceph.com first_in_suite: false job_id: '7670212' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: rbd qos iops burst: 100 rbd qos iops limit: 50 rbd qos schedule tick min: 100 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4807 sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 sleep_before_teardown: 0 subset: 19/128 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTlAqt8qW3rT8l4Pup9o3z5Ecf2Xz0zXC0u1TlaaiJUuJxBT0cyUG1AXURr2Ozkjq7yNQF0iziAHr0b0wLYSNI= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLGhoKTLgdLjEc7YLkKyzQCpQvd1oV4QXaQPEPfcn9jd4Tn/m3hXf5yDsvBietgFbZawJSabv5YxEb0AsjpSsE= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-23_20:16:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T21:13:22.434 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/qa; will attempt to use it 2024-04-23T21:13:22.434 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/qa/tasks 2024-04-23T21:13:22.434 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T21:13:22.436 INFO:teuthology.task.internal:Checking packages... 2024-04-23T21:13:22.460 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8' 2024-04-23T21:13:22.460 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T21:13:22.460 INFO:teuthology.packaging:ref: None 2024-04-23T21:13:22.460 INFO:teuthology.packaging:tag: None 2024-04-23T21:13:22.460 INFO:teuthology.packaging:branch: main 2024-04-23T21:13:22.460 INFO:teuthology.packaging:sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-23T21:13:22.460 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-23T21:13:22.686 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3219.g18874190 2024-04-23T21:13:22.688 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T21:13:22.695 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T21:13:22.695 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T21:13:22.712 INFO:teuthology.task.internal:Saving configuration 2024-04-23T21:13:22.724 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T21:13:22.731 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T21:13:22.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 21:07:57.468491', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTlAqt8qW3rT8l4Pup9o3z5Ecf2Xz0zXC0u1TlaaiJUuJxBT0cyUG1AXURr2Ozkjq7yNQF0iziAHr0b0wLYSNI='} 2024-04-23T21:13:22.776 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 21:07:57.470436', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLGhoKTLgdLjEc7YLkKyzQCpQvd1oV4QXaQPEPfcn9jd4Tn/m3hXf5yDsvBietgFbZawJSabv5YxEb0AsjpSsE='} 2024-04-23T21:13:22.777 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T21:13:22.783 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-04-23T21:13:22.783 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-04-23T21:13:22.783 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T21:13:22.853 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdbcd0cc3a0>, signals=[15]) 2024-04-23T21:13:22.853 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T21:13:22.860 INFO:teuthology.task.internal:Opening connections... 2024-04-23T21:13:22.860 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:13:22.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T21:13:22.940 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:13:22.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T21:13:23.013 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T21:13:23.020 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-23T21:13:23.037 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-23T21:13:23.037 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T21:13:23.094 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-23T21:13:23.095 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T21:13:23.095 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T21:13:23.095 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T21:13:23.095 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-23T21:13:23.119 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-04-23T21:13:23.136 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-04-23T21:13:23.136 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-04-23T21:13:23.193 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T21:13:23.194 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T21:13:23.195 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-04-23T21:13:23.218 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T21:13:23.228 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T21:13:23.233 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T21:13:23.233 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-23T21:13:23.236 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-04-23T21:13:23.252 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T21:13:23.258 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T21:13:23.258 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-23T21:13:23.292 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-04-23T21:13:23.506 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T21:13:23.554 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T21:13:23.568 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-23T21:13:23.568 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-23T21:13:23.568 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T21:13:23.569 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T21:13:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:13:23.585 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-04-23T21:13:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:13:23.587 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-23T21:13:23.643 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-437.el9.x86_64 2024-04-23T21:13:23.644 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-437.el9.x86_64 2024-04-23T21:13:23.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-04-23T21:13:23.645 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-437.el9.x86_64 2024-04-23T21:13:23.645 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-437.el9.x86_64 2024-04-23T21:13:23.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-04-23T21:13:25.924 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-23T21:13:26.255 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-04-23T21:13:36.365 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 2.0 MB/s | 19 MB 00:09 2024-04-23T21:13:40.891 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-23T21:13:43.101 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-23T21:13:44.318 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-23T21:13:49.309 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-23T21:13:49.786 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 867 kB/s | 19 MB 00:22 2024-04-23T21:13:51.183 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T21:13:51.183 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T21:13:51.255 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:13:51.256 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:13:51.256 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:13:51.376 DEBUG:teuthology.orchestra.run.smithi192:> echo no | sudo yum reinstall kernel || true 2024-04-23T21:13:51.868 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:02 ago on Tue 23 Apr 2024 09:13:49 PM UTC. 2024-04-23T21:13:51.994 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:13:52.006 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.006 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalling: 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 255 k 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 0 2024-04-23T21:13:52.007 INFO:teuthology.orchestra.run.smithi192.stderr:Operation aborted. 2024-04-23T21:13:52.069 INFO:teuthology.orchestra.run.smithi192.stdout:Is this ok [y/N]: 2024-04-23T21:13:52.069 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum reinstall -y kernel || true 2024-04-23T21:13:52.560 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 09:13:49 PM UTC. 2024-04-23T21:13:52.677 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:13:52.678 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.678 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-23T21:13:52.678 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.678 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalling: 2024-04-23T21:13:52.678 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 255 k 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 0 2024-04-23T21:13:52.679 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:13:52.986 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.4 MB/s | 255 kB 00:00 2024-04-23T21:13:52.987 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:13:52.987 INFO:teuthology.orchestra.run.smithi192.stdout:Total 828 kB/s | 255 kB 00:00 2024-04-23T21:13:53.009 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:13:53.194 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:13:53.195 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:13:53.199 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:13:53.200 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:13:53.348 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:13:53.438 INFO:teuthology.orchestra.run.smithi192.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T21:13:53.438 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:13:53.995 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-23T21:13:54.174 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:13:54.174 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T21:13:54.849 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:13:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalled: 2024-04-23T21:13:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T21:13:54.850 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:13:54.850 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:13:55.025 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T21:13:55.114 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T21:13:55.115 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T21:13:55.115 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T21:13:55.115 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-23T21:13:55.115 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T21:13:55.153 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-04-23T21:13:55.171 DEBUG:teuthology.parallel:result is None 2024-04-23T21:13:55.688 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-23T21:13:57.400 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-23T21:14:02.364 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-23T21:14:04.287 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T21:14:04.287 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T21:14:04.382 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:14:04.382 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:14:04.383 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:14:04.511 DEBUG:teuthology.orchestra.run.smithi111:> echo no | sudo yum reinstall kernel || true 2024-04-23T21:14:05.044 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 09:14:02 PM UTC. 2024-04-23T21:14:05.185 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout:Reinstalling: 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:14:05.192 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.193 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:05.193 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 255 k 2024-04-23T21:14:05.193 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 0 2024-04-23T21:14:05.193 INFO:teuthology.orchestra.run.smithi111.stderr:Operation aborted. 2024-04-23T21:14:05.248 INFO:teuthology.orchestra.run.smithi111.stdout:Is this ok [y/N]: 2024-04-23T21:14:05.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum reinstall -y kernel || true 2024-04-23T21:14:05.738 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 09:14:02 PM UTC. 2024-04-23T21:14:05.853 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:14:05.854 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.854 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-04-23T21:14:05.854 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.854 INFO:teuthology.orchestra.run.smithi111.stdout:Reinstalling: 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 255 k 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 0 2024-04-23T21:14:05.855 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:14:07.116 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-437.el9.x86_64.rpm 233 kB/s | 255 kB 00:01 2024-04-23T21:14:07.116 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:14:07.116 INFO:teuthology.orchestra.run.smithi111.stdout:Total 202 kB/s | 255 kB 00:01 2024-04-23T21:14:07.132 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:14:07.327 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:14:07.327 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:14:07.332 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:14:07.333 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:14:07.470 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:14:07.550 INFO:teuthology.orchestra.run.smithi111.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T21:14:07.550 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:14:08.243 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:14:08.243 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout:Reinstalled: 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:14:08.961 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:14:09.127 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T21:14:09.173 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T21:14:09.173 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T21:14:09.173 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T21:14:09.173 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-23T21:14:09.173 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T21:14:09.210 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-04-23T21:14:09.228 DEBUG:teuthology.parallel:result is None 2024-04-23T21:14:09.228 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T21:14:09.235 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T21:14:09.235 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T21:14:09.270 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T21:14:09.288 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T21:14:09.294 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T21:14:09.304 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T21:14:09.305 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T21:14:09.325 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T21:14:09.576 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T21:14:09.583 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T21:14:09.583 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T21:14:09.586 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T21:14:09.614 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T21:14:09.620 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T21:14:09.623 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T21:14:09.631 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T21:14:09.632 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T21:14:09.639 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T21:14:09.639 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T21:14:09.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T21:14:09.699 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T21:14:09.708 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T21:14:09.708 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T21:14:09.731 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T21:14:09.754 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T21:14:09.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T21:14:09.869 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T21:14:09.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T21:14:09.987 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:14:09.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T21:14:10.051 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T21:14:10.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T21:14:10.144 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T21:14:10.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T21:14:10.265 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:14:10.265 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T21:14:10.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-23T21:14:10.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-04-23T21:14:10.363 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T21:14:10.421 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T21:14:10.813 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T21:14:10.821 INFO:teuthology.task.internal:Starting timer... 2024-04-23T21:14:10.821 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T21:14:10.836 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T21:14:10.844 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-23T21:14:10.844 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-23T21:14:10.901 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-23T21:14:10.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-04-23T21:14:10.952 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-04-23T21:14:10.954 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T21:14:10.954 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-23T21:14:10.977 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-23T21:14:10.977 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-04-23T21:14:10.997 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-04-23T21:14:10.997 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-04-23T21:14:10.998 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T21:14:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:14:11.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T21:14:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:14:11.064 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T21:14:11.064 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-23T21:14:11.121 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-04-23T21:14:11.153 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T21:14:11.165 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T21:14:11.184 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T21:14:11.185 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T21:17:37.095 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-04-23T21:17:37.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-23T21:17:37.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T21:17:37.176 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-23T21:17:37.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-23T21:17:37.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-23T21:17:37.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T21:17:37.338 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-23T21:17:37.413 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-23T21:17:37.413 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T21:17:37.431 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T21:17:37.431 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T21:17:37.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T21:17:37.434 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T21:17:37.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T21:17:37.459 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T21:17:37.472 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T21:17:37.480 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T21:17:37.493 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T21:17:37.495 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-23T21:17:37.505 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-23T21:17:37.515 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-04-23T21:17:37.518 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T21:17:37.525 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-04-23T21:17:37.531 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T21:17:37.539 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T21:17:37.552 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T21:17:37.585 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-23T21:17:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T21:17:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-23T21:17:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.588 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.589 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.613 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-04-23T21:17:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T21:17:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-23T21:17:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.617 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.618 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:17:37.618 INFO:teuthology.run_tasks:Running task install... 2024-04-23T21:17:37.629 DEBUG:teuthology.task.install:project ceph 2024-04-23T21:17:37.629 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'}} 2024-04-23T21:17:37.629 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'} 2024-04-23T21:17:37.630 INFO:teuthology.task.install:Using flavor: default 2024-04-23T21:17:37.635 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-23T21:17:37.635 INFO:teuthology.task.install:extra packages: [] 2024-04-23T21:17:37.636 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8', 'tag': None, 'wait_for_package': False} 2024-04-23T21:17:37.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-23T21:17:37.638 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8', 'tag': None, 'wait_for_package': False} 2024-04-23T21:17:37.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-23T21:17:37.783 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/ 2024-04-23T21:17:37.783 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3212.g8254cb5a 2024-04-23T21:17:37.875 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/ 2024-04-23T21:17:37.875 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3212.g8254cb5a 2024-04-23T21:17:38.045 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T21:17:38.045 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:17:38.045 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T21:17:38.070 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-23T21:17:38.070 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T21:17:38.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T21:17:38.203 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-23T21:17:38.203 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:17:38.203 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-23T21:17:38.209 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T21:17:38.229 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-23T21:17:38.229 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-23T21:17:38.273 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-04-23T21:17:38.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-04-23T21:17:38.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-23T21:17:38.367 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-23T21:17:38.430 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-04-23T21:17:38.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-04-23T21:17:38.499 INFO:teuthology.orchestra.run.smithi192.stdout:48 files removed 2024-04-23T21:17:38.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-04-23T21:17:38.657 INFO:teuthology.orchestra.run.smithi111.stdout:48 files removed 2024-04-23T21:17:38.683 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-04-23T21:17:39.333 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 153 kB/s | 83 kB 00:00 2024-04-23T21:17:39.374 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-04-23T21:17:39.684 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 38 kB/s | 12 kB 00:00 2024-04-23T21:17:39.775 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-04-23T21:17:40.110 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-23T21:17:40.192 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-23T21:17:40.693 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-23T21:17:40.709 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-04-23T21:18:02.575 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 936 kB/s | 19 MB 00:20 2024-04-23T21:18:03.359 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 902 kB/s | 19 MB 00:21 2024-04-23T21:18:07.602 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-23T21:18:08.525 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.2 MB 00:02 2024-04-23T21:18:09.228 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-23T21:18:10.295 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-23T21:18:10.677 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 16 kB 00:00 2024-04-23T21:18:12.878 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-23T21:18:15.371 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-23T21:18:17.530 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 11 M 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.4 M 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.1 M 2024-04-23T21:18:17.532 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 5.4 M 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 21 M 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 25 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 747 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 500 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 5.4 M 2024-04-23T21:18:17.533 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 45 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 135 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 162 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 320 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 298 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 100 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T21:18:17.534 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:Install 30 Packages 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 60 M 2024-04-23T21:18:17.535 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:18:18.019 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-23T21:18:18.447 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-radosgw-19.0.0-3212.g8254cb5a.el9. 14 MB/s | 11 MB 00:00 2024-04-23T21:18:18.489 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-selinux-19.0.0-3212.g8254cb5a.el9. 606 kB/s | 25 kB 00:00 2024-04-23T21:18:18.564 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): libcephfs2-19.0.0-3212.g8254cb5a.el9.x8 9.7 MB/s | 747 kB 00:00 2024-04-23T21:18:18.623 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): libradosstriper1-19.0.0-3212.g8254cb5a. 8.4 MB/s | 500 kB 00:00 2024-04-23T21:18:18.906 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): librgw2-19.0.0-3212.g8254cb5a.el9.x86_6 19 MB/s | 5.4 MB 00:00 2024-04-23T21:18:18.948 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): python3-ceph-argparse-19.0.0-3212.g8254 1.1 MB/s | 45 kB 00:00 2024-04-23T21:18:18.998 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-ceph-common-19.0.0-3212.g8254cb 2.6 MB/s | 135 kB 00:00 2024-04-23T21:18:19.041 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-cephfs-19.0.0-3212.g8254cb5a.el 3.8 MB/s | 162 kB 00:00 2024-04-23T21:18:19.090 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-rados-19.0.0-3212.g8254cb5a.el9 6.3 MB/s | 320 kB 00:00 2024-04-23T21:18:19.141 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-rbd-19.0.0-3212.g8254cb5a.el9. 5.8 MB/s | 298 kB 00:00 2024-04-23T21:18:19.191 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-rgw-19.0.0-3212.g8254cb5a.el9. 2.0 MB/s | 100 kB 00:00 2024-04-23T21:18:19.950 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 141 kB/s | 107 kB 00:00 2024-04-23T21:18:20.195 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 11 M 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-04-23T21:18:20.197 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.4 M 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.1 M 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 5.4 M 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 21 M 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 25 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 747 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-23T21:18:20.198 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 500 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 5.4 M 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 45 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 135 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 162 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 320 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 298 k 2024-04-23T21:18:20.199 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 100 k 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout:Install 30 Packages 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-04-23T21:18:20.200 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:20.201 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 60 M 2024-04-23T21:18:20.201 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:18:20.225 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): ceph-base-19.0.0-3212.g8254cb5a.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-04-23T21:18:20.958 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 62 kB/s | 45 kB 00:00 2024-04-23T21:18:20.983 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 155 kB/s | 160 kB 00:01 2024-04-23T21:18:22.009 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 285 kB/s | 292 kB 00:01 2024-04-23T21:18:22.326 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): ceph-common-19.0.0-3212.g8254cb5a.el9. 4.5 MB/s | 21 MB 00:04 2024-04-23T21:18:22.362 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-common-19.0.0-3212.g8254cb5a.el9.x 14 MB/s | 21 MB 00:01 2024-04-23T21:18:22.426 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 451 kB/s | 662 kB 00:01 2024-04-23T21:18:22.429 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-selinux-19.0.0-3212.g8254cb5a.el9. 380 kB/s | 25 kB 00:00 2024-04-23T21:18:22.571 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): libcephfs2-19.0.0-3212.g8254cb5a.el9.x8 5.2 MB/s | 747 kB 00:00 2024-04-23T21:18:22.629 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): libradosstriper1-19.0.0-3212.g8254cb5a. 8.4 MB/s | 500 kB 00:00 2024-04-23T21:18:22.885 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 48 kB/s | 42 kB 00:00 2024-04-23T21:18:22.996 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): librgw2-19.0.0-3212.g8254cb5a.el9.x86_6 15 MB/s | 5.4 MB 00:00 2024-04-23T21:18:23.060 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 59 kB/s | 36 kB 00:00 2024-04-23T21:18:23.080 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): ceph-base-19.0.0-3212.g8254cb5a.el9.x86 2.4 MB/s | 5.4 MB 00:02 2024-04-23T21:18:23.105 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-ceph-argparse-19.0.0-3212.g8254 416 kB/s | 45 kB 00:00 2024-04-23T21:18:23.139 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-ceph-common-19.0.0-3212.g8254cb 2.3 MB/s | 135 kB 00:00 2024-04-23T21:18:23.164 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-cephfs-19.0.0-3212.g8254cb5a.el 2.7 MB/s | 162 kB 00:00 2024-04-23T21:18:23.215 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-rados-19.0.0-3212.g8254cb5a.el 4.2 MB/s | 320 kB 00:00 2024-04-23T21:18:23.240 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-rbd-19.0.0-3212.g8254cb5a.el9. 3.9 MB/s | 298 kB 00:00 2024-04-23T21:18:23.282 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): python3-rgw-19.0.0-3212.g8254cb5a.el9. 1.5 MB/s | 100 kB 00:00 2024-04-23T21:18:23.310 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 224 kB/s | 221 kB 00:00 2024-04-23T21:18:23.536 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-04-23T21:18:23.540 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-04-23T21:18:23.565 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 565 kB/s | 160 kB 00:00 2024-04-23T21:18:23.582 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-23T21:18:23.691 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): ceph-radosgw-19.0.0-3212.g8254cb5a.el9 3.7 MB/s | 11 MB 00:02 2024-04-23T21:18:23.716 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-23T21:18:23.736 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 457 kB/s | 309 kB 00:00 2024-04-23T21:18:23.750 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-04-23T21:18:23.800 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-04-23T21:18:23.826 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-23T21:18:23.851 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 356 kB/s | 36 kB 00:00 2024-04-23T21:18:23.885 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 563 kB/s | 47 kB 00:00 2024-04-23T21:18:23.886 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 326 kB/s | 49 kB 00:00 2024-04-23T21:18:23.945 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 64 kB/s | 26 kB 00:00 2024-04-23T21:18:24.020 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 505 kB/s | 67 kB 00:00 2024-04-23T21:18:24.060 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-04-23T21:18:24.179 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-23T21:18:24.194 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-04-23T21:18:24.261 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 711 kB/s | 309 kB 00:00 2024-04-23T21:18:24.328 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-04-23T21:18:24.354 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-04-23T21:18:24.471 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-04-23T21:18:24.503 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.8 MB/s | 4.4 MB 00:00 2024-04-23T21:18:24.536 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-04-23T21:18:24.538 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-04-23T21:18:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 820 kB/s | 192 kB 00:00 2024-04-23T21:18:24.670 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-23T21:18:24.922 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): librbd1-19.0.0-3212.g8254cb5a.el9.x86_ 7.0 MB/s | 3.1 MB 00:00 2024-04-23T21:18:25.047 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): librados2-19.0.0-3212.g8254cb5a.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-04-23T21:18:25.053 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:18:25.053 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.0 MB/s | 60 MB 00:07 2024-04-23T21:18:25.087 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): librbd1-19.0.0-3212.g8254cb5a.el9.x86_ 6.0 MB/s | 3.1 MB 00:00 2024-04-23T21:18:25.154 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:18:25.190 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:18:25.190 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:18:25.254 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): librados2-19.0.0-3212.g8254cb5a.el9.x8 4.9 MB/s | 3.4 MB 00:00 2024-04-23T21:18:25.256 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:18:25.256 INFO:teuthology.orchestra.run.smithi111.stdout:Total 12 MB/s | 60 MB 00:05 2024-04-23T21:18:25.358 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:18:25.394 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:18:25.394 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:18:25.661 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:18:25.661 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:18:25.862 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:18:25.863 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:18:27.157 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:18:27.239 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-23T21:18:27.268 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-23T21:18:27.293 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-23T21:18:27.316 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-23T21:18:27.342 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-23T21:18:27.370 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:18:27.446 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-23T21:18:27.484 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-04-23T21:18:27.509 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-04-23T21:18:27.532 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-23T21:18:27.541 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-23T21:18:27.555 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-04-23T21:18:27.595 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-23T21:18:27.699 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-23T21:18:27.753 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-04-23T21:18:27.932 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-23T21:18:28.045 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:28.080 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-04-23T21:18:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:28.211 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:28.235 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 8/34 2024-04-23T21:18:28.263 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 9/34 2024-04-23T21:18:28.303 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:28.326 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 9/34 2024-04-23T21:18:28.363 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-23T21:18:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 8/34 2024-04-23T21:18:28.407 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9. 11/34 2024-04-23T21:18:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 9/34 2024-04-23T21:18:28.485 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 12/34 2024-04-23T21:18:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 13/34 2024-04-23T21:18:28.509 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 9/34 2024-04-23T21:18:28.538 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-04-23T21:18:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 13/34 2024-04-23T21:18:28.582 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9. 11/34 2024-04-23T21:18:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-23T21:18:28.668 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 12/34 2024-04-23T21:18:28.698 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 13/34 2024-04-23T21:18:28.699 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-23T21:18:28.731 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-23T21:18:28.766 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 13/34 2024-04-23T21:18:28.772 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-23T21:18:28.800 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-04-23T21:18:28.805 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-23T21:18:28.834 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-23T21:18:28.866 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-23T21:18:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-04-23T21:18:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-04-23T21:18:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-04-23T21:18:29.054 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-04-23T21:18:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-23T21:18:29.192 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-04-23T21:18:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 22/34 2024-04-23T21:18:29.241 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-04-23T21:18:29.319 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 22/34 2024-04-23T21:18:29.342 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 23/34 2024-04-23T21:18:29.378 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-23T21:18:29.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-04-23T21:18:29.596 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 22/34 2024-04-23T21:18:29.686 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 22/34 2024-04-23T21:18:29.717 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 23/34 2024-04-23T21:18:29.753 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-23T21:18:29.860 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-23T21:18:29.890 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 26/34 2024-04-23T21:18:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 26/34 2024-04-23T21:18:30.039 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 27/34 2024-04-23T21:18:30.068 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x8 28/34 2024-04-23T21:18:30.238 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-23T21:18:30.266 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 26/34 2024-04-23T21:18:30.328 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 26/34 2024-04-23T21:18:30.396 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 27/34 2024-04-23T21:18:30.426 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x8 28/34 2024-04-23T21:18:32.362 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:32.490 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 30/34 2024-04-23T21:18:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 30/34 2024-04-23T21:18:33.144 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T21:18:33.144 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T21:18:33.144 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 29/34 2024-04-23T21:18:33.311 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 30/34 2024-04-23T21:18:33.402 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 30/34 2024-04-23T21:18:33.402 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-23T21:18:33.402 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-23T21:18:33.402 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:33.429 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:42.601 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:42.601 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-04-23T21:18:42.601 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-04-23T21:18:42.601 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-04-23T21:18:42.601 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-04-23T21:18:42.602 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-04-23T21:18:42.602 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-04-23T21:18:42.602 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-04-23T21:18:42.602 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:42.686 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:42.686 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-04-23T21:18:42.686 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-04-23T21:18:42.686 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-04-23T21:18:42.686 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-04-23T21:18:42.687 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-04-23T21:18:42.687 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /root 2024-04-23T21:18:42.687 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-04-23T21:18:42.687 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:43.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/34 2024-04-23T21:18:43.627 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/34 2024-04-23T21:18:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:18:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T21:18:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T21:18:43.628 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T21:18:43.628 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:43.631 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-23T21:18:43.680 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/34 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/34 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-23T21:18:43.706 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:43.708 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-23T21:18:43.722 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-23T21:18:43.722 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:43.783 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-23T21:18:43.783 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:44.929 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 5/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 6/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9. 8/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x8 9/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 10/34 2024-04-23T21:18:44.930 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 11/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 12/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 13/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-23T21:18:44.931 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-23T21:18:44.932 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 5/34 2024-04-23T21:18:45.033 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_6 6/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 7/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9. 8/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x8 9/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 10/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 11/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 12/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 13/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-23T21:18:45.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-23T21:18:45.035 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-23T21:18:45.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 31/34 2024-04-23T21:18:45.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-23T21:18:45.036 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/34 2024-04-23T21:18:45.745 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:45.745 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:45.745 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-04-23T21:18:45.745 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.745 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T21:18:45.746 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T21:18:45.747 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:45.748 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:18:45.820 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.822 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-23T21:18:45.823 INFO:teuthology.orchestra.run.smithi111.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-23T21:18:45.824 INFO:teuthology.orchestra.run.smithi111.stdout: re2-1:20211101-3.el9.x86_64 2024-04-23T21:18:45.824 INFO:teuthology.orchestra.run.smithi111.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-23T21:18:45.824 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:45.824 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:18:46.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-04-23T21:18:46.126 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-04-23T21:18:46.534 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:31 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:18:46.657 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:18:46.657 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 49 M 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 163 k 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-04-23T21:18:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:18:46.659 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2024-04-23T21:18:46.659 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 211 M 2024-04-23T21:18:46.659 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:18:46.661 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:29 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:18:46.783 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 49 M 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:18:46.784 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 163 k 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-04-23T21:18:46.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:18:46.786 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 50 M 2024-04-23T21:18:46.786 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 211 M 2024-04-23T21:18:46.786 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:18:47.207 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.0 MB/s | 305 kB 00:00 2024-04-23T21:18:47.266 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-23T21:18:47.309 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-04-23T21:18:47.359 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-23T21:18:47.451 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): libcephsqlite-19.0.0-3212.g8254cb5a.el9. 308 kB/s | 163 kB 00:00 2024-04-23T21:18:47.603 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): libcephsqlite-19.0.0-3212.g8254cb5a.el9. 302 kB/s | 163 kB 00:00 2024-04-23T21:18:53.519 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): ceph-test-19.0.0-3212.g8254cb5a.el9.x86_ 7.6 MB/s | 49 MB 00:06 2024-04-23T21:18:53.520 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:18:53.520 INFO:teuthology.orchestra.run.smithi111.stdout:Total 7.4 MB/s | 50 MB 00:06 2024-04-23T21:18:53.545 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:18:53.564 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:18:53.564 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:18:53.937 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:18:53.937 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:18:53.958 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): ceph-test-19.0.0-3212.g8254cb5a.el9.x86_ 7.0 MB/s | 49 MB 00:07 2024-04-23T21:18:53.960 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:18:53.960 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.8 MB/s | 50 MB 00:07 2024-04-23T21:18:53.984 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:18:54.003 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:18:54.004 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:18:54.373 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:18:54.374 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:18:54.458 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:18:54.536 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-23T21:18:54.584 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-23T21:18:54.612 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/4 2024-04-23T21:18:54.898 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:18:54.974 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-04-23T21:18:55.015 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-04-23T21:18:55.043 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/4 2024-04-23T21:18:59.165 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/4 2024-04-23T21:18:59.250 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/4 2024-04-23T21:18:59.667 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/4 2024-04-23T21:18:59.799 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/4 2024-04-23T21:19:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/4 2024-04-23T21:19:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/4 2024-04-23T21:19:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/4 2024-04-23T21:19:01.625 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-23T21:19:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-23T21:19:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:01.889 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:01.889 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T21:19:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T21:19:01.890 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:01.890 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/4 2024-04-23T21:19:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/4 2024-04-23T21:19:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/4 2024-04-23T21:19:02.157 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-04-23T21:19:02.174 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-04-23T21:19:02.455 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:02.456 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:02.678 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:45 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:02.767 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-04-23T21:19:02.802 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 6.6 k 2024-04-23T21:19:02.805 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 2.4 M 2024-04-23T21:19:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 1.5 M 2024-04-23T21:19:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 249 k 2024-04-23T21:19:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 4.7 M 2024-04-23T21:19:02.806 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 17 M 2024-04-23T21:19:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T21:19:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T21:19:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T21:19:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T21:19:02.807 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T21:19:02.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T21:19:02.809 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:02.810 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:02.810 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:02.810 INFO:teuthology.orchestra.run.smithi111.stdout:Install 36 Packages 2024-04-23T21:19:02.810 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:02.810 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 30 M 2024-04-23T21:19:02.811 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 107 M 2024-04-23T21:19:02.811 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:03.281 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:48 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:03.401 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 6.6 k 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 2.4 M 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 1.5 M 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 249 k 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 4.7 M 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 17 M 2024-04-23T21:19:03.404 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-23T21:19:03.405 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-23T21:19:03.406 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout:Install 36 Packages 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2024-04-23T21:19:03.407 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 107 M 2024-04-23T21:19:03.408 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:03.796 INFO:teuthology.orchestra.run.smithi111.stdout:(1/36): ceph-19.0.0-3212.g8254cb5a.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-04-23T21:19:03.846 INFO:teuthology.orchestra.run.smithi111.stdout:(2/36): ceph-mgr-19.0.0-3212.g8254cb5a.el9.x86_ 3.0 MB/s | 1.5 MB 00:00 2024-04-23T21:19:04.192 INFO:teuthology.orchestra.run.smithi192.stdout:(1/36): ceph-19.0.0-3212.g8254cb5a.el9.x86_64.r 20 kB/s | 6.6 kB 00:00 2024-04-23T21:19:04.471 INFO:teuthology.orchestra.run.smithi111.stdout:(3/36): ceph-mds-19.0.0-3212.g8254cb5a.el9.x86_ 2.1 MB/s | 2.4 MB 00:01 2024-04-23T21:19:04.597 INFO:teuthology.orchestra.run.smithi111.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3212.g8254 1.9 MB/s | 249 kB 00:00 2024-04-23T21:19:04.730 INFO:teuthology.orchestra.run.smithi111.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-04-23T21:19:04.814 INFO:teuthology.orchestra.run.smithi111.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-04-23T21:19:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-23T21:19:04.897 INFO:teuthology.orchestra.run.smithi111.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-04-23T21:19:04.939 INFO:teuthology.orchestra.run.smithi111.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-04-23T21:19:04.981 INFO:teuthology.orchestra.run.smithi111.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-04-23T21:19:05.209 INFO:teuthology.orchestra.run.smithi192.stdout:(2/36): ceph-mgr-19.0.0-3212.g8254cb5a.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-04-23T21:19:05.260 INFO:teuthology.orchestra.run.smithi192.stdout:(3/36): ceph-mds-19.0.0-3212.g8254cb5a.el9.x86_ 1.7 MB/s | 2.4 MB 00:01 2024-04-23T21:19:05.410 INFO:teuthology.orchestra.run.smithi192.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3212.g8254 1.6 MB/s | 249 kB 00:00 2024-04-23T21:19:05.577 INFO:teuthology.orchestra.run.smithi192.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-04-23T21:19:05.665 INFO:teuthology.orchestra.run.smithi111.stdout:(11/36): ceph-osd-19.0.0-3212.g8254cb5a.el9.x86 9.3 MB/s | 17 MB 00:01 2024-04-23T21:19:05.690 INFO:teuthology.orchestra.run.smithi111.stdout:(12/36): lua-5.4.4-4.el9.x86_64.rpm 266 kB/s | 188 kB 00:00 2024-04-23T21:19:05.790 INFO:teuthology.orchestra.run.smithi111.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-04-23T21:19:05.844 INFO:teuthology.orchestra.run.smithi192.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 4.6 MB/s | 1.2 MB 00:00 2024-04-23T21:19:05.894 INFO:teuthology.orchestra.run.smithi192.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-23T21:19:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 437 kB/s | 51 kB 00:00 2024-04-23T21:19:05.953 INFO:teuthology.orchestra.run.smithi192.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-04-23T21:19:06.003 INFO:teuthology.orchestra.run.smithi192.stdout:(9/36): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-04-23T21:19:06.007 INFO:teuthology.orchestra.run.smithi111.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 416 kB/s | 42 kB 00:00 2024-04-23T21:19:06.112 INFO:teuthology.orchestra.run.smithi192.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-04-23T21:19:06.174 INFO:teuthology.orchestra.run.smithi111.stdout:(16/36): python3-mako-1.1.4-6.el9.noarch.rpm 338 kB/s | 172 kB 00:00 2024-04-23T21:19:06.195 INFO:teuthology.orchestra.run.smithi192.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-23T21:19:06.279 INFO:teuthology.orchestra.run.smithi192.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-23T21:19:06.329 INFO:teuthology.orchestra.run.smithi192.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 695 kB/s | 35 kB 00:00 2024-04-23T21:19:06.379 INFO:teuthology.orchestra.run.smithi192.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-23T21:19:06.430 INFO:teuthology.orchestra.run.smithi192.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-04-23T21:19:06.538 INFO:teuthology.orchestra.run.smithi192.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-04-23T21:19:06.566 INFO:teuthology.orchestra.run.smithi111.stdout:(17/36): ceph-mon-19.0.0-3212.g8254cb5a.el9.x86 1.7 MB/s | 4.7 MB 00:02 2024-04-23T21:19:06.592 INFO:teuthology.orchestra.run.smithi111.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 362 kB/s | 151 kB 00:00 2024-04-23T21:19:06.617 INFO:teuthology.orchestra.run.smithi111.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-04-23T21:19:06.742 INFO:teuthology.orchestra.run.smithi111.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-23T21:19:06.780 INFO:teuthology.orchestra.run.smithi192.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 626 kB/s | 151 kB 00:00 2024-04-23T21:19:06.817 INFO:teuthology.orchestra.run.smithi111.stdout:(21/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2024-04-23T21:19:06.838 INFO:teuthology.orchestra.run.smithi192.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 748 kB/s | 43 kB 00:00 2024-04-23T21:19:06.843 INFO:teuthology.orchestra.run.smithi111.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-23T21:19:06.901 INFO:teuthology.orchestra.run.smithi111.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-04-23T21:19:06.922 INFO:teuthology.orchestra.run.smithi192.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-23T21:19:06.935 INFO:teuthology.orchestra.run.smithi111.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-04-23T21:19:06.985 INFO:teuthology.orchestra.run.smithi111.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-04-23T21:19:07.022 INFO:teuthology.orchestra.run.smithi192.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-04-23T21:19:07.027 INFO:teuthology.orchestra.run.smithi111.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-04-23T21:19:07.052 INFO:teuthology.orchestra.run.smithi111.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 824 kB/s | 358 kB 00:00 2024-04-23T21:19:07.064 INFO:teuthology.orchestra.run.smithi192.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-23T21:19:07.078 INFO:teuthology.orchestra.run.smithi111.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 501 kB/s | 46 kB 00:00 2024-04-23T21:19:07.106 INFO:teuthology.orchestra.run.smithi192.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-04-23T21:19:07.148 INFO:teuthology.orchestra.run.smithi192.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-04-23T21:19:07.161 INFO:teuthology.orchestra.run.smithi111.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-23T21:19:07.186 INFO:teuthology.orchestra.run.smithi111.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 495 kB/s | 79 kB 00:00 2024-04-23T21:19:07.190 INFO:teuthology.orchestra.run.smithi192.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-04-23T21:19:07.212 INFO:teuthology.orchestra.run.smithi111.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-04-23T21:19:07.232 INFO:teuthology.orchestra.run.smithi192.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2024-04-23T21:19:07.254 INFO:teuthology.orchestra.run.smithi111.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 973 kB/s | 90 kB 00:00 2024-04-23T21:19:07.282 INFO:teuthology.orchestra.run.smithi192.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-04-23T21:19:07.287 INFO:teuthology.orchestra.run.smithi111.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 355 kB/s | 36 kB 00:00 2024-04-23T21:19:07.321 INFO:teuthology.orchestra.run.smithi111.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-23T21:19:07.380 INFO:teuthology.orchestra.run.smithi111.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-04-23T21:19:07.405 INFO:teuthology.orchestra.run.smithi111.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-04-23T21:19:07.407 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:07.408 INFO:teuthology.orchestra.run.smithi111.stdout:Total 6.6 MB/s | 30 MB 00:04 2024-04-23T21:19:07.466 INFO:teuthology.orchestra.run.smithi192.stdout:(27/36): ceph-osd-19.0.0-3212.g8254cb5a.el9.x86 7.5 MB/s | 17 MB 00:02 2024-04-23T21:19:07.500 INFO:teuthology.orchestra.run.smithi192.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 363 kB/s | 79 kB 00:00 2024-04-23T21:19:07.547 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:07.567 INFO:teuthology.orchestra.run.smithi192.stdout:(29/36): ceph-mon-19.0.0-3212.g8254cb5a.el9.x86 1.4 MB/s | 4.7 MB 00:03 2024-04-23T21:19:07.586 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:07.586 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:07.592 INFO:teuthology.orchestra.run.smithi192.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-04-23T21:19:07.642 INFO:teuthology.orchestra.run.smithi192.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 719 kB/s | 36 kB 00:00 2024-04-23T21:19:07.709 INFO:teuthology.orchestra.run.smithi192.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-23T21:19:07.735 INFO:teuthology.orchestra.run.smithi192.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-04-23T21:19:07.760 INFO:teuthology.orchestra.run.smithi192.stdout:(34/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 467 kB/s | 90 kB 00:00 2024-04-23T21:19:07.785 INFO:teuthology.orchestra.run.smithi192.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-04-23T21:19:07.810 INFO:teuthology.orchestra.run.smithi192.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-04-23T21:19:07.813 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:07.813 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.8 MB/s | 30 MB 00:04 2024-04-23T21:19:07.877 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:07.879 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:07.953 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:07.993 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:07.993 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:08.290 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:08.291 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:09.147 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:09.199 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-23T21:19:09.232 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-23T21:19:09.260 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-23T21:19:09.331 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-23T21:19:09.400 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-23T21:19:09.433 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-23T21:19:09.526 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-23T21:19:09.554 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:09.603 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-23T21:19:09.605 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-23T21:19:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-23T21:19:09.643 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-23T21:19:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-23T21:19:09.679 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-23T21:19:09.709 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-23T21:19:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-23T21:19:09.743 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-23T21:19:09.777 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-23T21:19:09.781 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-23T21:19:09.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-23T21:19:09.818 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-23T21:19:09.852 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-23T21:19:09.887 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-23T21:19:09.904 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-23T21:19:09.925 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-23T21:19:09.975 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-23T21:19:09.999 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-23T21:19:10.016 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-23T21:19:10.051 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-23T21:19:10.081 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-23T21:19:10.124 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-23T21:19:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-23T21:19:10.158 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-23T21:19:10.198 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-23T21:19:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-23T21:19:10.233 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-23T21:19:10.267 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-23T21:19:10.278 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-23T21:19:10.305 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-23T21:19:10.317 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-23T21:19:10.365 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-23T21:19:10.378 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-23T21:19:10.422 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-23T21:19:10.516 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-23T21:19:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-23T21:19:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-23T21:19:10.735 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-23T21:19:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-23T21:19:10.800 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-23T21:19:10.836 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-23T21:19:10.838 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-23T21:19:10.870 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-23T21:19:10.886 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-23T21:19:10.942 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-23T21:19:10.997 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-23T21:19:11.129 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-23T21:19:11.159 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9. 31/36 2024-04-23T21:19:11.175 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-23T21:19:11.197 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/36 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/36 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T21:19:11.223 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:11.244 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-23T21:19:11.333 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-23T21:19:11.374 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-23T21:19:11.491 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-23T21:19:11.667 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9. 31/36 2024-04-23T21:19:11.694 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/36 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 32/36 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-23T21:19:11.719 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:12.656 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/36 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/36 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T21:19:12.683 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:13.124 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 34/36 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 34/36 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T21:19:13.147 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:13.161 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/36 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 33/36 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-23T21:19:13.187 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:13.366 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 35/36 2024-04-23T21:19:13.388 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 35/36 2024-04-23T21:19:13.388 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:13.389 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T21:19:13.389 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T21:19:13.389 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T21:19:13.389 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:13.452 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 36/36 2024-04-23T21:19:13.628 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 34/36 2024-04-23T21:19:13.651 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 34/36 2024-04-23T21:19:13.651 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:13.651 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-23T21:19:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T21:19:13.652 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-23T21:19:13.652 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:13.871 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 35/36 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 35/36 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-23T21:19:13.894 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:13.957 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 36/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 36/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 5/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9. 6/36 2024-04-23T21:19:15.131 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-23T21:19:15.132 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-23T21:19:15.133 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-23T21:19:15.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 36/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 3/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 4/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 5/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9. 6/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-23T21:19:15.638 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-23T21:19:15.639 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-23T21:19:15.640 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:16.017 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T21:19:16.018 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T21:19:16.019 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:16.020 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:16.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-04-23T21:19:16.564 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-23T21:19:16.564 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:16.564 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:16.564 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-23T21:19:16.565 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-23T21:19:16.566 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:16.567 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:16.884 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:59 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:16.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-04-23T21:19:16.963 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:19:17.004 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:17.005 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:19:17.005 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:17.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-04-23T21:19:17.447 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:02 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:17.526 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:19:17.567 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:17.568 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:19:17.568 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:17.590 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:00 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:17.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-04-23T21:19:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:17.709 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 762 k 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:17.710 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 762 k 2024-04-23T21:19:17.711 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 769 k 2024-04-23T21:19:17.711 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:17.998 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-19.0.0-3212.g8254cb5a.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-04-23T21:19:17.999 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:17.999 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-04-23T21:19:17.999 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:18.004 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:18.004 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:18.010 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:18.011 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:18.079 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:18.146 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:03 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:18.267 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 762 k 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:18.268 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 762 k 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 769 k 2024-04-23T21:19:18.269 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:18.561 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-19.0.0-3212.g8254cb5a.el9.noarch.rpm 2.5 MB/s | 762 kB 00:00 2024-04-23T21:19:18.562 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:18.562 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.5 MB/s | 762 kB 00:00 2024-04-23T21:19:18.562 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:18.567 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:18.567 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:18.573 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:18.573 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:18.643 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:18.659 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:18.969 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:19.142 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:19.143 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:19.180 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.380 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-04-23T21:19:19.524 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/1 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:19.783 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:19.901 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:02 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:19.991 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-04-23T21:19:20.024 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 145 k 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 145 k 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 435 k 2024-04-23T21:19:20.025 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:20.338 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-19.0.0-3212.g8254cb 463 kB/s | 145 kB 00:00 2024-04-23T21:19:20.339 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:20.339 INFO:teuthology.orchestra.run.smithi111.stdout:Total 461 kB/s | 145 kB 00:00 2024-04-23T21:19:20.339 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:20.347 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:20.347 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:20.403 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:20.403 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:20.513 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:05 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:20.555 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:20.637 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 145 k 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:20.638 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 145 k 2024-04-23T21:19:20.639 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 435 k 2024-04-23T21:19:20.639 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:20.647 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:20.672 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:20.672 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T21:19:20.672 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:20.942 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-19.0.0-3212.g8254cb 478 kB/s | 145 kB 00:00 2024-04-23T21:19:20.943 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:20.943 INFO:teuthology.orchestra.run.smithi192.stdout:Total 475 kB/s | 145 kB 00:00 2024-04-23T21:19:20.943 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:20.951 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:20.951 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:21.007 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:21.007 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:21.158 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:21.227 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:21.252 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:21.252 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:21.252 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-23T21:19:21.252 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:21.499 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:21.499 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:21.499 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:21.500 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:21.500 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:21.500 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:21.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-04-23T21:19:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a. 1/1 2024-04-23T21:19:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:22.112 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:22.112 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:19:22.113 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:22.113 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:22.217 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:05 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:22.297 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:19:22.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-04-23T21:19:22.339 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:22.340 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:19:22.340 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:22.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-04-23T21:19:22.838 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:07 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:22.911 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:05 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:22.918 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:19:22.960 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:22.962 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:19:22.962 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:23.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-04-23T21:19:23.031 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 3.5 M 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 34 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 16 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:23.033 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-04-23T21:19:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:23.034 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 8.6 M 2024-04-23T21:19:23.034 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 104 M 2024-04-23T21:19:23.034 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:23.525 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:08 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:23.646 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 3.5 M 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 34 k 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 16 k 2024-04-23T21:19:23.647 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-04-23T21:19:23.648 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:23.649 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 8.6 M 2024-04-23T21:19:23.649 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 104 M 2024-04-23T21:19:23.649 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:23.808 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3212.g82 140 kB/s | 34 kB 00:00 2024-04-23T21:19:23.825 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3212.g825 63 kB/s | 16 kB 00:00 2024-04-23T21:19:24.513 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): ceph-grafana-dashboards-19.0.0-3212.g82 124 kB/s | 34 kB 00:00 2024-04-23T21:19:24.805 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): ceph-prometheus-alerts-19.0.0-3212.g825 29 kB/s | 16 kB 00:00 2024-04-23T21:19:24.809 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 274 kB/s | 269 kB 00:00 2024-04-23T21:19:24.830 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.2 MB/s | 1.0 MB 00:00 2024-04-23T21:19:24.972 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-04-23T21:19:25.006 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-04-23T21:19:25.298 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): ceph-mgr-dashboard-19.0.0-3212.g8254cb5 3.3 MB/s | 3.5 MB 00:01 2024-04-23T21:19:25.398 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-04-23T21:19:25.643 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-04-23T21:19:25.726 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 539 kB/s | 1.0 MB 00:01 2024-04-23T21:19:25.765 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 692 kB/s | 548 kB 00:00 2024-04-23T21:19:25.784 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): ceph-mgr-dashboard-19.0.0-3212.g8254cb5 1.6 MB/s | 3.5 MB 00:02 2024-04-23T21:19:25.790 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 392 kB/s | 144 kB 00:00 2024-04-23T21:19:25.890 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 246 kB/s | 31 kB 00:00 2024-04-23T21:19:25.909 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-04-23T21:19:26.015 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 838 kB/s | 188 kB 00:00 2024-04-23T21:19:26.077 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-04-23T21:19:26.152 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 597 kB/s | 144 kB 00:00 2024-04-23T21:19:26.177 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 307 kB/s | 31 kB 00:00 2024-04-23T21:19:26.274 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.1 MB/s | 2.0 MB 00:00 2024-04-23T21:19:26.279 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:26.279 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.3 MB/s | 8.6 MB 00:02 2024-04-23T21:19:26.326 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:26.343 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:26.343 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:26.361 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-04-23T21:19:26.386 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 807 kB/s | 188 kB 00:00 2024-04-23T21:19:26.388 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:26.389 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.6 MB/s | 8.6 MB 00:03 2024-04-23T21:19:26.441 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:26.459 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:26.459 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:26.499 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:26.500 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:26.614 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:26.614 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:27.020 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:27.100 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-23T21:19:27.135 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-23T21:19:27.139 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:27.181 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-23T21:19:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-04-23T21:19:27.212 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-23T21:19:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-04-23T21:19:27.285 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-04-23T21:19:27.289 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-23T21:19:27.324 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-04-23T21:19:27.400 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-04-23T21:19:27.429 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-23T21:19:27.514 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-23T21:19:27.532 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-04-23T21:19:27.566 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-23T21:19:27.595 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-23T21:19:27.627 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-04-23T21:19:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9 10/12 2024-04-23T21:19:27.670 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-04-23T21:19:27.702 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-04-23T21:19:27.731 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9 10/12 2024-04-23T21:19:28.671 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el 11/12 2024-04-23T21:19:28.702 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 12/12 2024-04-23T21:19:28.775 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el 11/12 2024-04-23T21:19:28.806 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 12/12 2024-04-23T21:19:30.076 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 12/12 2024-04-23T21:19:30.076 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el 1/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 2/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9 3/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-23T21:19:30.077 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-23T21:19:30.078 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-23T21:19:30.078 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-23T21:19:30.078 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-23T21:19:30.078 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 12/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el 1/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noa 2/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9 3/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-04-23T21:19:30.134 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-04-23T21:19:30.135 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-04-23T21:19:30.135 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-04-23T21:19:30.378 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-23T21:19:30.378 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:30.378 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:30.378 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T21:19:30.378 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:30.379 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-23T21:19:30.417 INFO:teuthology.orchestra.run.smithi111.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:30.418 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:30.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-23T21:19:30.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-23T21:19:31.172 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:16 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:31.186 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:14 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:31.302 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:31.303 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 7.4 M 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T21:19:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-04-23T21:19:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:31.306 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 38 M 2024-04-23T21:19:31.306 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 209 M 2024-04-23T21:19:31.306 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:31.307 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 7.4 M 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-23T21:19:31.308 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout:======================================================================================= 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout:Install 12 Packages 2024-04-23T21:19:31.309 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:31.310 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 38 M 2024-04-23T21:19:31.310 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 209 M 2024-04-23T21:19:31.310 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:31.908 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-23T21:19:32.008 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-04-23T21:19:32.012 INFO:teuthology.orchestra.run.smithi111.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-23T21:19:32.058 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-04-23T21:19:32.087 INFO:teuthology.orchestra.run.smithi111.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-23T21:19:32.112 INFO:teuthology.orchestra.run.smithi111.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-04-23T21:19:32.150 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-04-23T21:19:32.154 INFO:teuthology.orchestra.run.smithi111.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 434 kB/s | 18 kB 00:00 2024-04-23T21:19:32.196 INFO:teuthology.orchestra.run.smithi111.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 845 kB/s | 35 kB 00:00 2024-04-23T21:19:32.226 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-04-23T21:19:32.293 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-23T21:19:32.343 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-23T21:19:32.396 INFO:teuthology.orchestra.run.smithi111.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-04-23T21:19:32.447 INFO:teuthology.orchestra.run.smithi111.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-04-23T21:19:32.493 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-32 9.7 MB/s | 7.4 MB 00:00 2024-04-23T21:19:32.635 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-04-23T21:19:32.694 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 10 MB/s | 4.7 MB 00:00 2024-04-23T21:19:32.713 INFO:teuthology.orchestra.run.smithi111.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.2 MB/s | 4.7 MB 00:00 2024-04-23T21:19:32.789 INFO:teuthology.orchestra.run.smithi111.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-04-23T21:19:32.814 INFO:teuthology.orchestra.run.smithi111.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-04-23T21:19:32.852 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-23T21:19:33.464 INFO:teuthology.orchestra.run.smithi111.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 4.6 MB/s | 7.4 MB 00:01 2024-04-23T21:19:33.631 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-04-23T21:19:33.631 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:33.631 INFO:teuthology.orchestra.run.smithi192.stdout:Total 16 MB/s | 38 MB 00:02 2024-04-23T21:19:33.765 INFO:teuthology.orchestra.run.smithi111.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-23T21:19:33.765 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:33.766 INFO:teuthology.orchestra.run.smithi111.stdout:Total 15 MB/s | 38 MB 00:02 2024-04-23T21:19:33.976 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:34.003 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:34.003 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:34.116 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:34.142 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:34.142 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:34.409 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:34.409 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:34.549 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:34.550 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:35.243 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:35.299 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-23T21:19:35.342 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-23T21:19:35.419 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:35.440 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-23T21:19:35.486 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-23T21:19:35.501 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-23T21:19:35.529 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-23T21:19:35.627 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-23T21:19:35.672 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-23T21:19:35.678 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-23T21:19:35.709 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-23T21:19:35.786 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-23T21:19:35.849 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-23T21:19:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-23T21:19:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-23T21:19:36.428 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-23T21:19:36.554 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-23T21:19:36.595 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-23T21:19:36.708 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-23T21:19:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-23T21:19:37.929 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-23T21:19:38.810 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-23T21:19:38.836 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 12/12 2024-04-23T21:19:38.972 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-23T21:19:39.005 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 12/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 12/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 1/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-23T21:19:41.451 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-23T21:19:41.452 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-23T21:19:41.452 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-23T21:19:41.452 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-23T21:19:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 12/12 2024-04-23T21:19:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254c 1/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-23T21:19:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-23T21:19:41.877 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-23T21:19:41.877 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T21:19:41.878 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:41.879 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-23T21:19:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:41.893 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-23T21:19:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:41.895 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:42.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-04-23T21:19:42.256 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-04-23T21:19:42.775 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:25 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:42.776 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:27 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:42.896 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:42.896 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 49 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T21:19:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-04-23T21:19:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:42.899 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.2 M 2024-04-23T21:19:42.899 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2024-04-23T21:19:42.899 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 49 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-23T21:19:42.900 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 2.2 M 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 26 M 2024-04-23T21:19:42.901 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:43.380 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 94 kB/s | 26 kB 00:00 2024-04-23T21:19:43.405 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 64 kB/s | 19 kB 00:00 2024-04-23T21:19:43.430 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): ceph-mgr-rook-19.0.0-3212.g8254cb5a.el9 153 kB/s | 49 kB 00:00 2024-04-23T21:19:43.530 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): ceph-mgr-rook-19.0.0-3212.g8254cb5a.el9 260 kB/s | 49 kB 00:00 2024-04-23T21:19:43.597 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 75 kB/s | 19 kB 00:00 2024-04-23T21:19:43.623 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 91 kB/s | 26 kB 00:00 2024-04-23T21:19:43.747 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-04-23T21:19:43.798 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 532 kB/s | 222 kB 00:00 2024-04-23T21:19:43.856 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 494 kB/s | 54 kB 00:00 2024-04-23T21:19:43.939 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 464 kB/s | 159 kB 00:00 2024-04-23T21:19:44.039 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 536 kB/s | 54 kB 00:00 2024-04-23T21:19:44.056 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 445 kB/s | 279 kB 00:00 2024-04-23T21:19:44.064 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 416 kB/s | 222 kB 00:00 2024-04-23T21:19:44.081 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-04-23T21:19:44.107 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 359 kB/s | 15 kB 00:00 2024-04-23T21:19:44.115 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-04-23T21:19:44.132 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 547 kB/s | 279 kB 00:00 2024-04-23T21:19:44.166 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-04-23T21:19:44.216 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.8 MB/s | 206 kB 00:00 2024-04-23T21:19:44.241 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-04-23T21:19:44.266 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-04-23T21:19:44.291 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 392 kB/s | 59 kB 00:00 2024-04-23T21:19:44.300 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-04-23T21:19:44.300 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:44.300 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-04-23T21:19:44.364 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:44.378 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:44.379 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:44.391 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-04-23T21:19:44.416 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2024-04-23T21:19:44.441 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-google-auth-2.29.0-1.el9.noarc 535 kB/s | 206 kB 00:00 2024-04-23T21:19:44.444 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:44.444 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-04-23T21:19:44.461 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:44.462 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:44.509 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:44.525 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:44.525 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:44.607 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:44.608 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:44.970 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:45.140 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-23T21:19:45.193 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-23T21:19:45.239 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:45.249 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-23T21:19:45.286 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-23T21:19:45.327 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-23T21:19:45.406 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-23T21:19:45.422 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-23T21:19:45.467 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-23T21:19:45.532 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-23T21:19:45.532 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-23T21:19:45.569 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-23T21:19:45.587 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-23T21:19:45.610 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-23T21:19:45.704 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-23T21:19:45.821 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-04-23T21:19:45.879 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-23T21:19:46.299 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-23T21:19:46.352 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-23T21:19:46.439 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-23T21:19:46.493 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-23T21:19:46.513 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 13/13 2024-04-23T21:19:46.591 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-23T21:19:46.642 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-23T21:19:46.676 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-23T21:19:46.724 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-23T21:19:46.746 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 13/13 2024-04-23T21:19:47.158 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 13/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 1/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-23T21:19:47.159 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-23T21:19:47.406 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 13/13 2024-04-23T21:19:47.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 1/13 2024-04-23T21:19:47.406 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-23T21:19:47.407 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-23T21:19:47.467 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-23T21:19:47.467 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:47.467 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T21:19:47.468 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:47.469 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:47.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-04-23T21:19:47.714 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-23T21:19:47.714 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:47.714 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:47.714 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:47.715 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:48.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-04-23T21:19:48.225 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:33 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:48.345 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:48.346 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:48.346 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:48.346 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:48.346 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 142 k 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-04-23T21:19:48.347 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:48.348 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 7.0 M 2024-04-23T21:19:48.348 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 33 M 2024-04-23T21:19:48.348 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:48.506 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:31 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:48.626 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 142 k 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-23T21:19:48.627 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:48.628 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 7.0 M 2024-04-23T21:19:48.629 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 33 M 2024-04-23T21:19:48.629 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:49.087 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-04-23T21:19:49.236 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3212.g8254cb5a.e 364 kB/s | 142 kB 00:00 2024-04-23T21:19:49.318 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-04-23T21:19:49.478 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 239 kB/s | 58 kB 00:00 2024-04-23T21:19:49.527 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3212.g8254cb5a.e 319 kB/s | 142 kB 00:00 2024-04-23T21:19:49.537 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-04-23T21:19:49.562 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 610 kB/s | 51 kB 00:00 2024-04-23T21:19:49.627 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-04-23T21:19:49.761 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-04-23T21:19:49.811 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 203 kB/s | 58 kB 00:00 2024-04-23T21:19:49.861 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 218 kB/s | 51 kB 00:00 2024-04-23T21:19:49.862 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:49.862 INFO:teuthology.orchestra.run.smithi111.stdout:Total 5.7 MB/s | 7.0 MB 00:01 2024-04-23T21:19:49.934 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:49.945 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:49.945 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:50.041 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:50.041 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:50.304 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.1 MB/s | 6.0 MB 00:01 2024-04-23T21:19:50.306 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:50.307 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-04-23T21:19:50.345 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:50.376 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:50.388 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:50.388 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:50.486 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:50.487 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:50.501 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-23T21:19:50.554 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-23T21:19:50.794 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-23T21:19:51.039 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-23T21:19:51.075 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-23T21:19:51.157 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-23T21:19:51.228 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-23T21:19:51.258 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 6/6 2024-04-23T21:19:51.555 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-23T21:19:51.641 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-23T21:19:51.705 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-23T21:19:51.734 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 6/6 2024-04-23T21:19:51.984 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 6/6 2024-04-23T21:19:51.984 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/6 2024-04-23T21:19:51.985 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-23T21:19:51.985 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-23T21:19:51.985 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-23T21:19:51.985 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-23T21:19:52.230 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-23T21:19:52.231 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:52.231 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:52.231 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:52.231 INFO:teuthology.orchestra.run.smithi111.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T21:19:52.231 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T21:19:52.232 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T21:19:52.232 INFO:teuthology.orchestra.run.smithi111.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T21:19:52.232 INFO:teuthology.orchestra.run.smithi111.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T21:19:52.232 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:52.232 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:52.465 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 6/6 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 1/6 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-23T21:19:52.505 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-23T21:19:52.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-23T21:19:52.766 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:52.766 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:52.767 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:52.983 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:35 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:53.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-04-23T21:19:53.103 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 890 k 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:53.105 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-04-23T21:19:53.105 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:53.105 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 970 k 2024-04-23T21:19:53.105 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 2.8 M 2024-04-23T21:19:53.105 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:53.413 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 660 kB/s | 80 kB 00:00 2024-04-23T21:19:53.536 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:38 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:53.657 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:53.657 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:53.657 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:53.657 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:53.657 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 890 k 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-04-23T21:19:53.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:53.659 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 970 k 2024-04-23T21:19:53.659 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.8 M 2024-04-23T21:19:53.659 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:53.755 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-fuse-19.0.0-3212.g8254cb5a.el9.x86_ 1.9 MB/s | 890 kB 00:00 2024-04-23T21:19:53.756 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:53.756 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.5 MB/s | 970 kB 00:00 2024-04-23T21:19:53.767 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:53.789 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:53.789 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:53.867 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:53.867 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:54.049 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:54.140 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 407 kB/s | 80 kB 00:00 2024-04-23T21:19:54.191 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-23T21:19:54.270 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/2 2024-04-23T21:19:54.883 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/2 2024-04-23T21:19:54.884 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/2 2024-04-23T21:19:55.147 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-23T21:19:55.147 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:55.148 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:55.148 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-23T21:19:55.148 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:55.148 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:55.367 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-volume 2024-04-23T21:19:55.532 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-19.0.0-3212.g8254cb5a.el9.x86_ 560 kB/s | 890 kB 00:01 2024-04-23T21:19:55.532 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:55.533 INFO:teuthology.orchestra.run.smithi192.stdout:Total 518 kB/s | 970 kB 00:01 2024-04-23T21:19:55.546 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:55.571 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:55.572 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:55.653 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:55.653 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:55.836 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:55.879 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:38 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:55.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-23T21:19:56.002 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:56.003 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:56.003 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 273 k 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 350 k 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 1.5 M 2024-04-23T21:19:56.004 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:56.047 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/2 2024-04-23T21:19:56.237 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 668 kB/s | 77 kB 00:00 2024-04-23T21:19:56.479 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): ceph-volume-19.0.0-3212.g8254cb5a.el9.no 764 kB/s | 273 kB 00:00 2024-04-23T21:19:56.480 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:56.480 INFO:teuthology.orchestra.run.smithi111.stdout:Total 735 kB/s | 350 kB 00:00 2024-04-23T21:19:56.491 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:56.501 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:56.501 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:56.526 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:56.526 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:56.667 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2/2 2024-04-23T21:19:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/2 2024-04-23T21:19:56.850 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-23T21:19:56.882 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2/2 2024-04-23T21:19:56.896 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2/2 2024-04-23T21:19:56.897 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:56.897 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T21:19:56.897 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:56.925 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-23T21:19:56.925 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:56.925 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:56.925 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-23T21:19:56.925 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:56.926 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:57.203 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2024-04-23T21:19:57.560 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 1/2 2024-04-23T21:19:57.727 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:19:57.848 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-23T21:19:57.848 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:57.848 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:19:57.849 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:57.849 INFO:teuthology.orchestra.run.smithi111.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T21:19:57.849 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:57.849 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:19:57.851 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:19:57.851 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:57.851 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-3212.g8254cb5a.el9 ceph-noarch 273 k 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-04-23T21:19:57.852 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:57.853 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 350 k 2024-04-23T21:19:57.853 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.5 M 2024-04-23T21:19:57.853 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:19:58.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-04-23T21:19:58.229 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): ceph-volume-19.0.0-3212.g8254cb5a.el9.no 1.1 MB/s | 273 kB 00:00 2024-04-23T21:19:58.346 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 217 kB/s | 77 kB 00:00 2024-04-23T21:19:58.347 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:58.347 INFO:teuthology.orchestra.run.smithi192.stdout:Total 708 kB/s | 350 kB 00:00 2024-04-23T21:19:58.358 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:19:58.368 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:19:58.368 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:19:58.395 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:19:58.395 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:19:58.541 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:19:58.681 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:41 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:19:58.728 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-23T21:19:58.766 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2/2 2024-04-23T21:19:58.781 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2/2 2024-04-23T21:19:58.781 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:19:58.781 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-23T21:19:58.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:58.805 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:19:58.805 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 126 k 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:19:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:19:58.807 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 126 k 2024-04-23T21:19:58.807 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 456 k 2024-04-23T21:19:58.807 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:19:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-19.0.0-3212.g8254cb5a.el9.x86_64 367 kB/s | 126 kB 00:00 2024-04-23T21:19:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:19:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:Total 365 kB/s | 126 kB 00:00 2024-04-23T21:19:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:19:59.159 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:19:59.159 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:19:59.222 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:19:59.222 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:19:59.381 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:19:59.432 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 1/2 2024-04-23T21:19:59.474 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:19:59.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-23T21:19:59.698 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:59.698 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:19:59.699 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-3212.g8254cb5a.el9.noarch 2024-04-23T21:19:59.699 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-23T21:19:59.699 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:19:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:19:59.894 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:00.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-04-23T21:20:00.139 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:00.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:00.139 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:20:00.139 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:00.140 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:00.140 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:00.388 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-04-23T21:20:00.547 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:45 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:00.667 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:00.668 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-04-23T21:20:00.668 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 126 k 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 126 k 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2024-04-23T21:20:00.669 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:20:00.879 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-19.0.0-3212.g8254cb5a.el9.x86_64 605 kB/s | 126 kB 00:00 2024-04-23T21:20:00.879 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:00.880 INFO:teuthology.orchestra.run.smithi192.stdout:Total 600 kB/s | 126 kB 00:00 2024-04-23T21:20:00.880 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:20:00.885 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:20:00.885 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:20:00.918 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:43 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:00.950 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:20:00.950 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:20:00.999 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:01.041 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:01.042 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:01.042 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:01.114 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:20:01.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-04-23T21:20:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:01.636 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:44 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:01.682 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:01.756 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:01.756 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:01.756 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-04-23T21:20:01.756 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 32 k 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:01.757 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 32 k 2024-04-23T21:20:01.758 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 142 k 2024-04-23T21:20:01.758 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:20:01.940 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:01.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:01.940 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:20:01.941 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:01.941 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:01.941 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:02.107 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-19.0.0-3212.g8254cb5a.el9.x86_6 91 kB/s | 32 kB 00:00 2024-04-23T21:20:02.107 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:02.107 INFO:teuthology.orchestra.run.smithi111.stdout:Total 90 kB/s | 32 kB 00:00 2024-04-23T21:20:02.108 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:20:02.111 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:20:02.111 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:20:02.113 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:20:02.114 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:20:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:20:02.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-04-23T21:20:02.310 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:02.682 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:02.739 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:47 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:02.820 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:02.860 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:02.861 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:02.862 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:02.896 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:20:02.896 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:02.897 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:02.897 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:02.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-04-23T21:20:03.066 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-04-23T21:20:03.451 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:48 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:03.570 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:03.571 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:03.571 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-04-23T21:20:03.571 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:03.571 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 32 k 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 32 k 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 142 k 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:20:03.572 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:46 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:03.650 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:03.691 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:03.692 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:03.692 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:03.742 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-19.0.0-3212.g8254cb5a.el9.x86_6 186 kB/s | 32 kB 00:00 2024-04-23T21:20:03.743 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:03.743 INFO:teuthology.orchestra.run.smithi192.stdout:Total 185 kB/s | 32 kB 00:00 2024-04-23T21:20:03.743 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:20:03.747 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:20:03.747 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:20:03.749 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:20:03.749 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:20:03.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-04-23T21:20:03.827 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:20:03.939 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:04.273 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:47 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:04.281 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:04.351 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:04.392 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:04.393 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:04.394 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:04.453 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:04.496 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:04.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-04-23T21:20:04.956 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:47 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:05.035 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:05.075 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:05.076 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:05.076 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:05.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-04-23T21:20:05.204 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:05.283 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:05.323 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:05.324 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:05.324 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:05.392 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-04-23T21:20:05.645 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:48 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:05.725 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:05.766 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:05.767 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:05.767 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:05.828 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-04-23T21:20:05.903 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:05.983 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:06.024 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:06.025 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:06.025 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:06.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-04-23T21:20:06.329 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:49 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:06.407 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:06.448 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:06.449 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:06.449 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:06.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-04-23T21:20:06.600 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:51 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:06.680 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:06.720 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:06.721 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:06.721 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:06.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-04-23T21:20:07.022 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:07.101 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:07.142 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-04-23T21:20:07.143 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:07.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-04-23T21:20:07.302 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:52 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:07.382 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:07.423 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:07.424 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:07.424 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:07.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-04-23T21:20:07.712 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:50 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:07.832 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 85 k 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 85 k 2024-04-23T21:20:07.834 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 231 k 2024-04-23T21:20:07.834 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:20:08.007 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:53 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:08.051 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 392 kB/s | 85 kB 00:00 2024-04-23T21:20:08.051 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:08.052 INFO:teuthology.orchestra.run.smithi111.stdout:Total 389 kB/s | 85 kB 00:00 2024-04-23T21:20:08.052 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:20:08.059 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:20:08.059 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:20:08.086 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:08.121 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:20:08.122 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:20:08.126 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:08.127 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:08.127 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:08.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-04-23T21:20:08.267 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:20:08.356 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:08.684 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:08.714 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:53 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:08.795 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 is already installed. 2024-04-23T21:20:08.837 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:08.838 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-23T21:20:08.838 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:08.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-04-23T21:20:08.914 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:08.915 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:08.915 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:20:08.915 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:08.915 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:08.915 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:09.185 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-04-23T21:20:09.423 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:54 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:09.542 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 85 k 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:09.543 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 85 k 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 231 k 2024-04-23T21:20:09.544 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:20:09.711 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:52 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:09.729 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 459 kB/s | 85 kB 00:00 2024-04-23T21:20:09.730 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:09.730 INFO:teuthology.orchestra.run.smithi192.stdout:Total 456 kB/s | 85 kB 00:00 2024-04-23T21:20:09.730 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:20:09.737 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:20:09.738 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:20:09.801 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:20:09.801 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:20:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:09.835 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:09.835 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.1 M 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.1 M 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 13 M 2024-04-23T21:20:09.836 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:20:09.951 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:20:10.053 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:10.263 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 7.2 MB/s | 3.1 MB 00:00 2024-04-23T21:20:10.264 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:10.264 INFO:teuthology.orchestra.run.smithi111.stdout:Total 7.2 MB/s | 3.1 MB 00:00 2024-04-23T21:20:10.264 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:20:10.274 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:20:10.274 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:20:10.357 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:20:10.358 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:20:10.453 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:10.729 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:10.730 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:10.730 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:20:10.730 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:10.730 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:10.730 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:20:10.843 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:10.872 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:10.873 INFO:teuthology.orchestra.run.smithi111.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:20:10.873 INFO:teuthology.orchestra.run.smithi111.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T21:20:10.873 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T21:20:10.873 INFO:teuthology.orchestra.run.smithi111.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T21:20:10.873 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:10.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-04-23T21:20:11.512 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:56 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:11.637 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:11.637 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:11.637 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 3.1 M 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.1 M 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 13 M 2024-04-23T21:20:11.638 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:11.963 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:12.199 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-04-23T21:20:12.292 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 4.7 MB/s | 3.1 MB 00:00 2024-04-23T21:20:12.292 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:12.293 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.7 MB/s | 3.1 MB 00:00 2024-04-23T21:20:12.293 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:20:12.302 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:20:12.302 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:20:12.383 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:20:12.383 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:20:12.716 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:55 ago on Tue 23 Apr 2024 09:18:17 PM UTC. 2024-04-23T21:20:12.807 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:20:12.839 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 170 k 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-04-23T21:20:12.840 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:12.841 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 170 k 2024-04-23T21:20:12.841 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 494 k 2024-04-23T21:20:12.841 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-23T21:20:12.876 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-23T21:20:12.901 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:13.083 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 703 kB/s | 170 kB 00:00 2024-04-23T21:20:13.084 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:13.084 INFO:teuthology.orchestra.run.smithi111.stdout:Total 699 kB/s | 170 kB 00:00 2024-04-23T21:20:13.084 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-23T21:20:13.091 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-23T21:20:13.092 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-23T21:20:13.154 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-23T21:20:13.155 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-23T21:20:13.308 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-23T21:20:13.405 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:13.903 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:14.034 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:14.034 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:14.034 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-23T21:20:14.034 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:14.034 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:14.035 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-23T21:20:14.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-04-23T21:20:14.278 DEBUG:teuthology.parallel:result is None 2024-04-23T21:20:14.655 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:59 ago on Tue 23 Apr 2024 09:18:15 PM UTC. 2024-04-23T21:20:14.777 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-23T21:20:14.778 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:14.778 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-23T21:20:14.778 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:14.778 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-3212.g8254cb5a.el9 ceph 170 k 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 170 k 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 494 k 2024-04-23T21:20:14.779 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-23T21:20:15.139 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-19.0.0-3212.g8254cb5a.el9.x86_64.rpm 474 kB/s | 170 kB 00:00 2024-04-23T21:20:15.139 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-23T21:20:15.140 INFO:teuthology.orchestra.run.smithi192.stdout:Total 472 kB/s | 170 kB 00:00 2024-04-23T21:20:15.140 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-23T21:20:15.147 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-23T21:20:15.147 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-23T21:20:15.210 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-23T21:20:15.210 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-23T21:20:15.372 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-23T21:20:15.467 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:15.871 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 1/1 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-3212.g8254cb5a.el9.x86_64 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:16.128 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-23T21:20:16.367 DEBUG:teuthology.parallel:result is None 2024-04-23T21:20:16.368 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-23T21:20:16.586 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T21:20:16.613 INFO:teuthology.orchestra.run.smithi111.stdout:19.0.0-3212.g8254cb5a.el9 2024-04-23T21:20:16.614 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3212.g8254cb5a.el9 2024-04-23T21:20:16.614 INFO:teuthology.task.install:The correct ceph version 19.0.0-3212.g8254cb5a is installed. 2024-04-23T21:20:16.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 2024-04-23T21:20:16.753 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-23T21:20:16.774 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-3212.g8254cb5a.el9 2024-04-23T21:20:16.774 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3212.g8254cb5a.el9 2024-04-23T21:20:16.774 INFO:teuthology.task.install:The correct ceph version 19.0.0-3212.g8254cb5a is installed. 2024-04-23T21:20:16.776 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-23T21:20:16.776 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:16.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T21:20:16.805 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:16.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-23T21:20:16.839 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-23T21:20:16.839 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:16.839 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T21:20:16.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T21:20:16.938 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:16.938 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-04-23T21:20:16.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-23T21:20:17.026 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-23T21:20:17.026 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:17.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T21:20:17.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T21:20:17.118 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:17.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-23T21:20:17.141 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-23T21:20:17.204 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-23T21:20:17.204 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:17.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T21:20:17.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T21:20:17.294 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:17.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-04-23T21:20:17.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-23T21:20:17.380 INFO:teuthology.run_tasks:Running task ceph... 2024-04-23T21:20:17.464 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-23T21:20:17.464 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /var/log/ceph 2024-04-23T21:20:17.467 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-04-23T21:20:17.489 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-23T21:20:17.489 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T21:20:17.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-23T21:20:17.554 INFO:tasks.ceph:Creating extra log directories... 2024-04-23T21:20:17.554 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T21:20:17.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-23T21:20:17.629 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-23T21:20:17.629 INFO:tasks.ceph:config {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-23T21:20:17.630 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212', 'branch': 'main', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7670212', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4807, 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8', 'sleep_before_teardown': 0, 'subset': '19/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f', 'targets': {'smithi111.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPTlAqt8qW3rT8l4Pup9o3z5Ecf2Xz0zXC0u1TlaaiJUuJxBT0cyUG1AXURr2Ozkjq7yNQF0iziAHr0b0wLYSNI=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJLGhoKTLgdLjEc7YLkKyzQCpQvd1oV4QXaQPEPfcn9jd4Tn/m3hXf5yDsvBietgFbZawJSabv5YxEb0AsjpSsE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-23_20:16:13', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-23T21:20:17.630 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T21:20:17.661 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-23T21:20:17.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T21:20:17.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-04-23T21:20:17.739 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:17.740 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T21:20:17.793 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T21:20:17.793 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-23T21:20:17.847 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-23 21:20:11.124150213 +0000 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-23 21:17:18.187208662 +0000 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-23 21:17:18.187208662 +0000 2024-04-23T21:20:17.848 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-23 21:17:18.187208662 +0000 2024-04-23T21:20:17.848 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T21:20:17.909 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-23T21:20:17.909 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-23T21:20:17.909 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000205508 s, 2.5 MB/s 2024-04-23T21:20:17.910 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T21:20:17.966 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-23T21:20:18.021 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T21:20:18.021 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-23 21:20:11.125150194 +0000 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-23 21:17:18.527201806 +0000 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-23 21:17:18.527201806 +0000 2024-04-23T21:20:18.022 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-23 21:17:18.527201806 +0000 2024-04-23T21:20:18.022 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T21:20:18.083 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-23T21:20:18.083 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-23T21:20:18.083 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000174342 s, 2.9 MB/s 2024-04-23T21:20:18.084 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T21:20:18.140 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-23 21:20:11.125150194 +0000 2024-04-23T21:20:18.194 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-23 21:17:18.872194848 +0000 2024-04-23T21:20:18.195 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-23 21:17:18.872194848 +0000 2024-04-23T21:20:18.195 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-23 21:17:18.872194848 +0000 2024-04-23T21:20:18.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T21:20:18.256 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-23T21:20:18.256 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-23T21:20:18.256 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000237846 s, 2.2 MB/s 2024-04-23T21:20:18.257 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T21:20:18.313 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-23T21:20:18.368 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-23 21:20:11.125150194 +0000 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-23 21:17:19.204188152 +0000 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-23 21:17:19.204188152 +0000 2024-04-23T21:20:18.369 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-23 21:17:19.204188152 +0000 2024-04-23T21:20:18.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T21:20:18.430 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-23T21:20:18.431 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-23T21:20:18.431 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.00023485 s, 2.2 MB/s 2024-04-23T21:20:18.432 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T21:20:18.487 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:18.487 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:18.487 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T21:20:18.506 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T21:20:18.506 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-04-23T21:20:18.564 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T21:20:18.564 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.564 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-23T21:20:18.564 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.565 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.565 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-23 21:20:13.144582003 +0000 2024-04-23T21:20:18.565 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-23 21:17:18.185020652 +0000 2024-04-23T21:20:18.565 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-23 21:17:18.185020652 +0000 2024-04-23T21:20:18.565 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-23 21:17:18.185020652 +0000 2024-04-23T21:20:18.565 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T21:20:18.631 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-23T21:20:18.631 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-23T21:20:18.631 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000225794 s, 2.3 MB/s 2024-04-23T21:20:18.632 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T21:20:18.691 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-23 21:20:13.145581985 +0000 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-23 21:17:18.516014130 +0000 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-23 21:17:18.516014130 +0000 2024-04-23T21:20:18.748 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-23 21:17:18.516014130 +0000 2024-04-23T21:20:18.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T21:20:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-23T21:20:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-23T21:20:18.813 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000368191 s, 1.4 MB/s 2024-04-23T21:20:18.814 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T21:20:18.872 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-04-23T21:20:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T21:20:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:18.928 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-23 21:20:13.145581985 +0000 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-23 21:17:18.856007431 +0000 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-23 21:17:18.856007431 +0000 2024-04-23T21:20:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-23 21:17:18.856007431 +0000 2024-04-23T21:20:18.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T21:20:18.991 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-23T21:20:18.992 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-23T21:20:18.992 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00019481 s, 2.6 MB/s 2024-04-23T21:20:18.993 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T21:20:19.050 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-23 21:20:13.145581985 +0000 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-23 21:17:19.215000357 +0000 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-23 21:17:19.215000357 +0000 2024-04-23T21:20:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-23 21:17:19.215000357 +0000 2024-04-23T21:20:19.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T21:20:19.169 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-23T21:20:19.169 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-23T21:20:19.169 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000243756 s, 2.1 MB/s 2024-04-23T21:20:19.170 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T21:20:19.226 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:19.226 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-23T21:20:19.226 INFO:tasks.ceph:Generating config... 2024-04-23T21:20:19.227 INFO:tasks.ceph:[client] rbd qos iops burst = 100 2024-04-23T21:20:19.227 INFO:tasks.ceph:[client] rbd qos iops limit = 50 2024-04-23T21:20:19.228 INFO:tasks.ceph:[client] rbd qos schedule tick min = 100 2024-04-23T21:20:19.228 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-23T21:20:19.228 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-23T21:20:19.228 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-23T21:20:19.228 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-23T21:20:19.228 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-23T21:20:19.228 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-23T21:20:19.228 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-23T21:20:19.228 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-23T21:20:19.228 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-23T21:20:19.229 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-23T21:20:19.229 INFO:tasks.ceph:Setting up mon.a... 2024-04-23T21:20:19.229 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-23T21:20:19.290 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.keyring 2024-04-23T21:20:19.293 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-23T21:20:19.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T21:20:19.401 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.111'), ('mon.c', '[v2:172.21.15.111:3301,v1:172.21.15.111:6790]'), ('mon.b', '172.21.15.192')] 2024-04-23T21:20:19.402 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790],172.21.15.192', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-23T21:20:19.403 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:19.403 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T21:20:19.457 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.111 --addv c '[v2:172.21.15.111:3301,v1:172.21.15.111:6790]' --add b 172.21.15.192 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:19.542 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:19.542 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: generated fsid 3556a001-ee44-4b6e-8d50-1bc45ce74758 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:setting min_mon_release = quincy 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 0 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 3556a001-ee44-4b6e-8d50-1bc45ce74758 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:last_changed 2024-04-23T21:20:19.543740+0000 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-04-23T21:20:19.543740+0000 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:min_mon_release 17 (quincy) 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:election_strategy: 1 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.a 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.b 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:2: [v2:172.21.15.111:3301/0,v1:172.21.15.111:6790/0] mon.c 2024-04-23T21:20:19.543 INFO:teuthology.orchestra.run.smithi111.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-23T21:20:19.545 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-23T21:20:19.560 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3556a001-ee44-4b6e-8d50-1bc45ce74758... 2024-04-23T21:20:19.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T21:20:19.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-23T21:20:19.639 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-04-23T21:20:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: chdir = "" 2024-04-23T21:20:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T21:20:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: auth supported = cephx 2024-04-23T21:20:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: filestore xattr use omap = true 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: mon clock drift allowed = 1.000 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: osd crush chooseleaf type = 0 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: auth debug = true 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on old message = true 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: ms die on bug = true 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: mon pg warn max object skew = 0 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T21:20:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default size = 2 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow primary affinity = true 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd allow pg remap = true 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on legacy crush tunables = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on crush straw calc version zero = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on no sortbitwise = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on osd down out interval zero = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on too few osds = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: mon_allow_pool_size_one = true 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T21:20:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: osd default data pool replay window = 5 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon allow pool delete = true 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon cluster log file level = debug 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: debug asserts on shutdown = true 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon health detail to clog = false 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon host = "172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790],172.21.15.192" 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon client directed command retry = 5 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: mon warn on pool no app = False 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: ms inject socket failures = 5000 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 3556a001-ee44-4b6e-8d50-1bc45ce74758 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout:[osd] 2024-04-23T21:20:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: osd journal size = 100 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub load threshold = 5.0 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd scrub max interval = 600 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd mclock profile = high_recovery_ops 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd recover clone overlap = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd recovery max chunk = 1048576 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug shutdown = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug op order = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug verify stray on activate = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug trim objects = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd open classes on start = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug pg log writeout = true 2024-04-23T21:20:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: osd deep scrub update digest min age = 30 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: osd map max advance = 10 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: journal zero on create = true 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: filestore ondisk finisher threads = 3 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: filestore apply finisher threads = 3 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: bdev debug aio = true 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: osd debug misdirected ops = true 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore block size = 96636764160 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore compression algorithm = snappy 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore compression mode = aggressive 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: bluestore fsck on mount = True 2024-04-23T21:20:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluefs = 1/20 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug bluestore = 1/20 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug osd = 20 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug rocksdb = 4/10 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd full ratio = 0.9 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd nearfull ratio = 0.8 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: osd blocked scrub grace period = 3600 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: osd failsafe full ratio = 0.95 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: osd max markdown count = 1000 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: osd objectstore = bluestore 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: osd shutdown pgref assert = True 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr] 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-04-23T21:20:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: debug mgr = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mgr/telemetry/nag = false 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout:[mon] 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug ms = 1 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug mon = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug paxos = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: debug auth = 20 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon data avail warn = 5 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon mgr mkfs grace = 240 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min pgs per osd = 4 2024-04-23T21:20:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd reporter subtree level = osd 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: mon osd prime pg temp = true 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: mon reweight min bytes per osd = 10 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: # 1m isn't quite enough 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: mon_down_mkfs_grace = 2m 2024-04-23T21:20:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: mon_warn_on_filestore_osds = false 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout:[client] 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rgw cache enabled = true 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable ops log = true 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rgw enable usage log = true 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rbd qos iops burst = 100 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rbd qos iops limit = 50 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: rbd qos schedule tick min = 100 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.a] 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.c] 2024-04-23T21:20:19.648 INFO:teuthology.orchestra.run.smithi111.stdout:[mon.b] 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.649 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.650 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-04-23T21:20:19.651 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.652 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-04-23T21:20:19.652 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.652 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-04-23T21:20:19.653 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-04-23T21:20:19.653 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790],172.21.15.192" 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = 3556a001-ee44-4b6e-8d50-1bc45ce74758 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-04-23T21:20:19.654 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug trim objects = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-04-23T21:20:19.655 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression algorithm = snappy 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression mode = aggressive 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2024-04-23T21:20:19.656 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: osd blocked scrub grace period = 3600 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: osd max markdown count = 1000 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: osd shutdown pgref assert = True 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-04-23T21:20:19.657 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-04-23T21:20:19.658 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-04-23T21:20:19.659 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout: rbd qos iops burst = 100 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout: rbd qos iops limit = 50 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout: rbd qos schedule tick min = 100 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2024-04-23T21:20:19.660 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2024-04-23T21:20:19.661 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-23T21:20:19.662 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-23T21:20:19.754 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-23T21:20:19.755 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:19.755 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-23T21:20:19.769 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:19.769 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-23T21:20:19.824 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:20:19.824 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:19.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T21:20:19.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T21:20:19.896 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:19.896 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:19.951 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:20:19.951 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:19.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-23T21:20:19.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-23T21:20:19.988 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:19.989 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:20.048 INFO:tasks.ceph:Setting up mon nodes... 2024-04-23T21:20:20.052 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-23T21:20:20.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-23T21:20:20.104 INFO:teuthology.orchestra.run.smithi111.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-23T21:20:20.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T21:20:20.174 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-23T21:20:20.177 INFO:tasks.ceph:Setting up mds nodes... 2024-04-23T21:20:20.177 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-23T21:20:20.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T21:20:20.216 INFO:teuthology.orchestra.run.smithi111.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-23T21:20:20.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-23T21:20:20.269 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-23T21:20:20.281 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-23T21:20:20.281 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi111.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-23T21:20:20.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-23T21:20:20.306 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:20.306 INFO:tasks.ceph:role: osd.0 2024-04-23T21:20:20.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:20:20.306 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T21:20:20.376 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:20.376 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:20.376 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:20.376 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:20.376 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:20.377 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:20.377 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:20.377 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:20.377 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:20.377 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:20.878 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-04-23T21:20:20.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-04-23T21:20:20.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-23T21:20:21.009 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-23T21:20:21.041 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-23T21:20:21.106 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:21.106 INFO:tasks.ceph:role: osd.1 2024-04-23T21:20:21.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:20:21.106 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T21:20:21.177 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:21.177 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:21.177 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:21.177 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:21.177 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:21.178 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:21.178 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:21.178 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:21.178 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:21.178 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:21.635 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-04-23T21:20:21.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-04-23T21:20:21.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-23T21:20:21.725 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-23T21:20:21.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-23T21:20:21.816 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:21.816 INFO:tasks.ceph:role: osd.2 2024-04-23T21:20:21.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:20:21.817 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:21.888 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:22.364 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-04-23T21:20:22.365 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-04-23T21:20:22.366 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-23T21:20:22.490 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-23T21:20:22.519 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-23T21:20:22.583 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:22.583 INFO:tasks.ceph:role: osd.3 2024-04-23T21:20:22.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:20:22.583 DEBUG:teuthology.orchestra.run.smithi111:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:22.653 INFO:teuthology.orchestra.run.smithi111.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:22.654 INFO:teuthology.orchestra.run.smithi111.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:22.654 INFO:teuthology.orchestra.run.smithi111.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:23.157 INFO:teuthology.orchestra.run.smithi111.stdout:Discarding blocks...Done. 2024-04-23T21:20:23.159 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi111.front.sepia.ceph.com -o noatime 2024-04-23T21:20:23.159 DEBUG:teuthology.orchestra.run.smithi111:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-23T21:20:23.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-23T21:20:23.342 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.428+0000 7fd995d99640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-23T21:20:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.428+0000 7fd995d99640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-23T21:20:23.429 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.428+0000 7fd995d99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T21:20:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.428+0000 7fd995d99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T21:20:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.428+0000 7fd995d99640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-23T21:20:23.430 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:23.429+0000 7fd995d99640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-23T21:20:26.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T21:20:26.310 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.397+0000 7f7a0aa30640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.398+0000 7f7a0aa30640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.398+0000 7f7a0aa30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.398+0000 7f7a0aa30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.398+0000 7f7a0aa30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-23T21:20:26.403 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:26.398+0000 7f7a0aa30640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-23T21:20:29.247 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T21:20:29.274 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.360+0000 7fd2e6bb8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.361+0000 7fd2e6bb8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.361+0000 7fd2e6bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.361+0000 7fd2e6bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.361+0000 7fd2e6bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-23T21:20:29.361 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:29.361+0000 7fd2e6bb8640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-23T21:20:32.210 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T21:20:32.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:32.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-23T21:20:32.326 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-23T21:20:32.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T21:20:32.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T21:20:32.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-23T21:20:32.327 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:20:32.326+0000 7f5c55aa0640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-23T21:20:35.177 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T21:20:35.204 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-23T21:20:35.233 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:35.233 INFO:tasks.ceph:role: osd.4 2024-04-23T21:20:35.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:20:35.233 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:35.308 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:35.820 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-23T21:20:35.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-23T21:20:35.822 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-23T21:20:35.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-23T21:20:35.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-23T21:20:36.060 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:36.060 INFO:tasks.ceph:role: osd.5 2024-04-23T21:20:36.060 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:20:36.060 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-23T21:20:36.131 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:36.131 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:36.131 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:36.131 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:36.131 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:36.132 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:36.132 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:36.132 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:36.132 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:36.132 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:36.572 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-23T21:20:36.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-23T21:20:36.575 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-23T21:20:36.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-23T21:20:36.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-23T21:20:36.758 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:36.758 INFO:tasks.ceph:role: osd.6 2024-04-23T21:20:36.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:20:36.758 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:36.830 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:37.323 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-23T21:20:37.326 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-23T21:20:37.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-23T21:20:37.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-23T21:20:37.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-23T21:20:37.543 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-23T21:20:37.543 INFO:tasks.ceph:role: osd.7 2024-04-23T21:20:37.543 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:20:37.543 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-23T21:20:37.614 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-04-23T21:20:37.615 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-23T21:20:37.615 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-23T21:20:37.615 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-23T21:20:37.615 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-23T21:20:38.142 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-04-23T21:20:38.144 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-04-23T21:20:38.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-23T21:20:38.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-23T21:20:38.299 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:38.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.396+0000 7f3498587640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-23T21:20:38.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.397+0000 7f3498587640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-23T21:20:38.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.397+0000 7f3498587640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T21:20:38.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.397+0000 7f3498587640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T21:20:38.396 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.397+0000 7f3498587640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-23T21:20:38.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:38.397+0000 7f3498587640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-23T21:20:41.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T21:20:41.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.365+0000 7f5895780640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-23T21:20:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.365+0000 7f5895780640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-23T21:20:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.365+0000 7f5895780640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T21:20:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.365+0000 7f5895780640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T21:20:41.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.365+0000 7f5895780640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-23T21:20:41.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:41.366+0000 7f5895780640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-23T21:20:44.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T21:20:44.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.323+0000 7f34eaa0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-23T21:20:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.323+0000 7f34eaa0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-23T21:20:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.323+0000 7f34eaa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T21:20:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.323+0000 7f34eaa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T21:20:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.323+0000 7f34eaa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-23T21:20:44.324 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:44.324+0000 7f34eaa0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-23T21:20:47.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T21:20:47.201 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.285+0000 7febd4efb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.285+0000 7febd4efb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.285+0000 7febd4efb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.285+0000 7febd4efb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.285+0000 7febd4efb640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-23T21:20:47.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:47.286+0000 7febd4efb640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-23T21:20:50.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T21:20:50.193 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-23T21:20:50.194 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.194 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-23T21:20:50.222 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.223 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-23T21:20:50.290 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-23T21:20:50.357 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.357 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-23T21:20:50.423 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-23T21:20:50.489 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-23T21:20:50.518 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.518 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-23T21:20:50.585 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-23T21:20:50.652 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-23T21:20:50.718 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.719 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-23T21:20:50.784 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:50.784 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-23T21:20:50.802 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:50.802 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-23T21:20:50.840 INFO:tasks.ceph:Adding keys to all mons... 2024-04-23T21:20:50.840 DEBUG:teuthology.orchestra.run.smithi111:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T21:20:50.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-23T21:20:50.884 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.y] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAUJihm7KtLBhAAGaHk8QZFpHmp9WcBHJwj+w== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.0] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAXJihm+0yiGRAAjifF80cVO8IPhyzKj1oP4w== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.1] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAaJihm66HOFxAASC/fw60BrvbrX53gtKewhw== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.2] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAdJihmWn2XFRAA2ddbU5WjJLUZtjAmZ8XF3A== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.3] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAgJihm6w+GExAA7Gpk5Xc+fYnDCOffgnPkoA== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[mgr.x] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAUJihm3ap6ChAAtoETuNoeC4VMCcvCNSFI1Q== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.4] 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAmJihmVHKtFxAAQOyP1zNotDTgqAGFXEb8KA== 2024-04-23T21:20:50.885 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.5] 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQApJihm46nXFRAAeDC8HUmOp1cQVwVxEm8zvw== 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.6] 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAsJihmSHZYExAAx8nZSItTH3EOtaYMSjiGww== 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout:[osd.7] 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAvJihmJMwdERAA+RsbiEAxgVcuuZQtIPyMJQ== 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAUJihmEkQADRAAnfuhCUKGK9H1TJgE8S/WZg== 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2024-04-23T21:20:50.886 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAUJihmTGsjEBAAuOn2zPLQeZvnd64pNpRgzA== 2024-04-23T21:20:50.905 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2024-04-23T21:20:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAUJihm7KtLBhAAGaHk8QZFpHmp9WcBHJwj+w== 2024-04-23T21:20:50.905 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-04-23T21:20:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAXJihm+0yiGRAAjifF80cVO8IPhyzKj1oP4w== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAaJihm66HOFxAASC/fw60BrvbrX53gtKewhw== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAdJihmWn2XFRAA2ddbU5WjJLUZtjAmZ8XF3A== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAgJihm6w+GExAA7Gpk5Xc+fYnDCOffgnPkoA== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAUJihm3ap6ChAAtoETuNoeC4VMCcvCNSFI1Q== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAmJihmVHKtFxAAQOyP1zNotDTgqAGFXEb8KA== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQApJihm46nXFRAAeDC8HUmOp1cQVwVxEm8zvw== 2024-04-23T21:20:50.906 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAsJihmSHZYExAAx8nZSItTH3EOtaYMSjiGww== 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAvJihmJMwdERAA+RsbiEAxgVcuuZQtIPyMJQ== 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAUJihmEkQADRAAnfuhCUKGK9H1TJgE8S/WZg== 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-04-23T21:20:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAUJihmTGsjEBAAuOn2zPLQeZvnd64pNpRgzA== 2024-04-23T21:20:50.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T21:20:50.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T21:20:51.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.103 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.153 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.204 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T21:20:51.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-23T21:20:51.255 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.356 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.358 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-23T21:20:51.457 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T21:20:51.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T21:20:51.511 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T21:20:51.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-23T21:20:51.564 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-23T21:20:51.565 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-23T21:20:51.587 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T21:20:51.681 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-23T21:20:51.704 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-23T21:20:51.768 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T21:20:51.862 DEBUG:teuthology.orchestra.run.smithi111:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-23T21:20:51.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-23T21:20:51.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-23T21:20:52.005 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-23T21:20:52.029 DEBUG:teuthology.orchestra.run.smithi111:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:52.031 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-23T21:20:52.084 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-23T21:20:52.084 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-23T21:20:52.084 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-23T21:20:52.086 INFO:tasks.ceph.mon.a:Started 2024-04-23T21:20:52.087 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-23T21:20:52.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-23T21:20:52.089 INFO:tasks.ceph.mon.c:Started 2024-04-23T21:20:52.089 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-23T21:20:52.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-23T21:20:52.127 INFO:tasks.ceph.mon.b:Started 2024-04-23T21:20:52.127 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-23T21:20:52.127 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-23T21:20:52.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-23T21:20:52.129 INFO:tasks.ceph.mgr.y:Started 2024-04-23T21:20:52.130 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-23T21:20:52.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-23T21:20:52.131 INFO:tasks.ceph.mgr.x:Started 2024-04-23T21:20:52.132 INFO:tasks.ceph:Setting crush tunables to default 2024-04-23T21:20:52.132 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd crush tunables default 2024-04-23T21:20:57.484 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2024-04-23T21:20:57.495 INFO:tasks.ceph:check_enable_crimson: False 2024-04-23T21:20:57.496 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-23T21:20:57.496 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:57.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-23T21:20:57.520 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:57.520 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-23T21:20:57.526 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:57.527+0000 7f4d24442040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T21:20:57.584 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:57.584 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-23T21:20:57.647 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-23T21:20:57.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-23T21:20:57.688 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:57.688+0000 7f4d24442040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T21:20:57.710 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:57.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-23T21:20:57.734 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:57.734 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-23T21:20:57.742 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:57.743+0000 7f4d24442040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T21:20:57.765 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:57.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-23T21:20:57.827 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-23T21:20:57.828 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-23T21:20:57.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 30c47d3d-583d-4987-9b86-247b80cb80ae 0 2024-04-23T21:20:58.002 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:58.003+0000 7f4d24442040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T21:20:58.050 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:58.050+0000 7f4d24442040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T21:20:58.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-04-23T21:20:58.234+0000 7f13b9aa8640 0 --2- 172.21.15.192:0/2732302924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f13b413db10 0x7f13b4141580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-23T21:20:58.320 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:58.321+0000 7fbfece3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T21:20:58.334 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-04-23T21:20:58.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new d7c695c5-c361-4c22-b4e7-bd16572f0550 1 2024-04-23T21:20:58.465 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:58.465+0000 7fbfece3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T21:20:58.517 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:58.517+0000 7fbfece3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T21:20:58.533 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-04-23T21:20:58.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 46e4a0e9-b403-4a09-92ca-57245c3fec30 2 2024-04-23T21:20:58.746 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-04-23T21:20:58.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new c18de0c4-3fd5-46a0-85a6-f813c8202ea8 3 2024-04-23T21:20:58.760 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:58.761+0000 7fbfece3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T21:20:58.804 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:58.805+0000 7fbfece3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T21:20:58.809 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:58.809+0000 7f4d24442040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T21:20:58.900 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:58.901+0000 7f4d24442040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T21:20:58.942 INFO:teuthology.orchestra.run.smithi192.stdout:3 2024-04-23T21:20:58.945 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:58.945+0000 7f4d24442040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T21:20:58.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new d0562ad9-c051-475a-a40b-248d222c8921 4 2024-04-23T21:20:59.120 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:59.120+0000 7f4d24442040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T21:20:59.141 INFO:teuthology.orchestra.run.smithi192.stdout:4 2024-04-23T21:20:59.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 344f8beb-b979-4248-92df-c34cd560be6a 5 2024-04-23T21:20:59.170 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:59.171+0000 7f4d24442040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T21:20:59.226 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:59.226+0000 7f4d24442040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T21:20:59.339 INFO:teuthology.orchestra.run.smithi192.stdout:5 2024-04-23T21:20:59.345 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:59.346+0000 7f4d24442040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T21:20:59.351 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 62434311-7dc6-4a3e-bd07-1e8915b37709 6 2024-04-23T21:20:59.497 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.498+0000 7fbfece3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T21:20:59.573 INFO:teuthology.orchestra.run.smithi192.stdout:6 2024-04-23T21:20:59.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new db260d92-d3d7-464e-ab34-0bedd2991992 7 2024-04-23T21:20:59.586 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.584+0000 7fbfece3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T21:20:59.625 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.626+0000 7fbfece3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T21:20:59.747 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:20:59.748+0000 7f4d24442040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T21:20:59.768 INFO:teuthology.orchestra.run.smithi192.stdout:7 2024-04-23T21:20:59.784 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-23T21:20:59.784 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-23T21:20:59.787 INFO:tasks.ceph.osd.0:Started 2024-04-23T21:20:59.787 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-23T21:20:59.787 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-23T21:20:59.789 INFO:tasks.ceph.osd.1:Started 2024-04-23T21:20:59.789 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-23T21:20:59.789 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-23T21:20:59.790 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.790+0000 7fbfece3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T21:20:59.791 INFO:tasks.ceph.osd.2:Started 2024-04-23T21:20:59.791 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-23T21:20:59.792 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-23T21:20:59.793 INFO:tasks.ceph.osd.3:Started 2024-04-23T21:20:59.794 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-23T21:20:59.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-23T21:20:59.796 INFO:tasks.ceph.osd.4:Started 2024-04-23T21:20:59.796 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-23T21:20:59.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-23T21:20:59.798 INFO:tasks.ceph.osd.5:Started 2024-04-23T21:20:59.798 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-23T21:20:59.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-23T21:20:59.800 INFO:tasks.ceph.osd.6:Started 2024-04-23T21:20:59.801 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-23T21:20:59.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-23T21:20:59.803 INFO:tasks.ceph.osd.7:Started 2024-04-23T21:20:59.804 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:20:59.868 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.869+0000 7fbfece3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T21:20:59.931 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:20:59.932+0000 7fbfece3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T21:20:59.989 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:20:59.989 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":10,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:20:59.765723+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T21:20:59.765723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:21:00.000 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-23T21:21:00.001 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-23T21:21:00.038 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.039+0000 7f4d24442040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T21:21:00.048 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:00.048+0000 7fbfece3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T21:21:00.089 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.089+0000 7f4d24442040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T21:21:00.207 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.207+0000 7f4d24442040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T21:21:00.390 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:00.391+0000 7fe279d68640 -1 Falling back to public interface 2024-04-23T21:21:00.397 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:00.397+0000 7fbcb5613640 -1 Falling back to public interface 2024-04-23T21:21:00.405 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:00.403+0000 7f8836a1d640 -1 Falling back to public interface 2024-04-23T21:21:00.406 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:00.405+0000 7fd10cb06640 -1 Falling back to public interface 2024-04-23T21:21:00.406 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:00.405+0000 7fbacf80c640 -1 Falling back to public interface 2024-04-23T21:21:00.414 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:00.415+0000 7ff3db433640 -1 Falling back to public interface 2024-04-23T21:21:00.420 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:00.420+0000 7f324356e640 -1 Falling back to public interface 2024-04-23T21:21:00.421 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:00.422+0000 7f3c7b71f640 -1 Falling back to public interface 2024-04-23T21:21:00.431 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:00.432+0000 7fbfece3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T21:21:00.486 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.486+0000 7f4d24442040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T21:21:00.533 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.533+0000 7f4d24442040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T21:21:00.687 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:00.688+0000 7fbfece3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T21:21:00.690 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.691+0000 7f4d24442040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T21:21:00.739 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:00.740+0000 7fbfece3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T21:21:00.808 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T21:21:00.826 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.827+0000 7f4d24442040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T21:21:00.857 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:00.858+0000 7fbfece3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T21:21:00.875 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.876+0000 7f4d24442040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T21:21:00.945 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.945+0000 7f4d24442040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T21:21:00.966 INFO:teuthology.misc.health.smithi111.stdout: 2024-04-23T21:21:00.966 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":10,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:20:59.765723+0000","last_up_change":"0.000000","last_in_change":"2024-04-23T21:20:59.765723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:21:00.976 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-23T21:21:00.991 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:00.992+0000 7f4d24442040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T21:21:01.137 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.138+0000 7fbfece3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T21:21:01.184 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.184+0000 7fbfece3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T21:21:01.339 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.340+0000 7fbfece3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T21:21:01.467 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:01.468+0000 7f4d24442040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T21:21:01.475 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.476+0000 7fbfece3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T21:21:01.526 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.526+0000 7fbfece3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T21:21:01.583 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:01.584+0000 7f4d24442040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T21:21:01.595 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.597+0000 7fbfece3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T21:21:01.645 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:01.646+0000 7fbfece3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T21:21:02.121 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:02.122+0000 7fbfece3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T21:21:02.237 INFO:tasks.ceph.mgr.y.smithi111.stderr:2024-04-23T21:21:02.238+0000 7fbfece3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T21:21:02.598 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:02.598+0000 7f4cc502d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-23T21:21:02.599 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-04-23T21:21:02.600+0000 7f4cc502d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-23T21:21:03.883 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:03.884+0000 7fbcb5613640 -1 osd.5 0 log_to_monitors true 2024-04-23T21:21:03.907 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:03.908+0000 7fbacf80c640 -1 osd.1 0 log_to_monitors true 2024-04-23T21:21:03.912 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:03.913+0000 7fe279d68640 -1 osd.2 0 log_to_monitors true 2024-04-23T21:21:03.924 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:03.926+0000 7f8836a1d640 -1 osd.0 0 log_to_monitors true 2024-04-23T21:21:03.934 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:03.935+0000 7f3c7b71f640 -1 osd.6 0 log_to_monitors true 2024-04-23T21:21:03.934 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:03.935+0000 7f324356e640 -1 osd.4 0 log_to_monitors true 2024-04-23T21:21:03.942 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:03.943+0000 7fd10cb06640 -1 osd.7 0 log_to_monitors true 2024-04-23T21:21:03.964 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:03.965+0000 7ff3db433640 -1 osd.3 0 log_to_monitors true 2024-04-23T21:21:06.692 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:06.692+0000 7f323f504640 -1 osd.4 0 waiting for initial osdmap 2024-04-23T21:21:06.699 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:06.700+0000 7fbacb1df640 -1 osd.1 0 waiting for initial osdmap 2024-04-23T21:21:06.700 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:06.701+0000 7fd1092b0640 -1 osd.7 0 waiting for initial osdmap 2024-04-23T21:21:06.704 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:06.705+0000 7fe276512640 -1 osd.2 0 waiting for initial osdmap 2024-04-23T21:21:06.711 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:06.712+0000 7f3237324640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.712 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:06.713+0000 7fbac2fff640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.713 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:06.714+0000 7fd1008bc640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.717 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:06.717+0000 7fbcb0faa640 -1 osd.5 0 waiting for initial osdmap 2024-04-23T21:21:06.717 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:06.717+0000 7f3c776b5640 -1 osd.6 0 waiting for initial osdmap 2024-04-23T21:21:06.718 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:06.719+0000 7fe26db1e640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.727 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:06.728+0000 7ff3d69c3640 -1 osd.3 0 waiting for initial osdmap 2024-04-23T21:21:06.732 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:06.733+0000 7fbca8dca640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.740 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:06.741+0000 7f3c6f4d5640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.742 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:06.743+0000 7f8832407640 -1 osd.0 0 waiting for initial osdmap 2024-04-23T21:21:06.745 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:06.746+0000 7ff3ce7e3640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:06.753 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:06.754+0000 7f882a238640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:07.783 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-23T21:21:08.077 INFO:teuthology.misc.health.smithi111.stdout: 2024-04-23T21:21:08.077 INFO:teuthology.misc.health.smithi111.stdout:{"epoch":13,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:21:07.708812+0000","last_up_change":"2024-04-23T21:21:07.708812+0000","last_in_change":"2024-04-23T21:20:59.765723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6817","nonce":1285376165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6819","nonce":1285376165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6823","nonce":1285376165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6821","nonce":1285376165}]},"public_addr":"172.21.15.111:6817/1285376165","cluster_addr":"172.21.15.111:6819/1285376165","heartbeat_back_addr":"172.21.15.111:6823/1285376165","heartbeat_front_addr":"172.21.15.111:6821/1285376165","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6801","nonce":2687705294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6803","nonce":2687705294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6807","nonce":2687705294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6805","nonce":2687705294}]},"public_addr":"172.21.15.192:6801/2687705294","cluster_addr":"172.21.15.192:6803/2687705294","heartbeat_back_addr":"172.21.15.192:6807/2687705294","heartbeat_front_addr":"172.21.15.192:6805/2687705294","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:21:08.089 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-23T21:21:08.090 INFO:tasks.ceph:Creating RBD pool 2024-04-23T21:21:08.090 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-23T21:21:08.714 INFO:teuthology.orchestra.run.smithi111.stderr:pool 'rbd' created 2024-04-23T21:21:08.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-23T21:21:09.728 INFO:teuthology.orchestra.run.smithi111.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-23T21:21:09.744 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-23T21:21:09.745 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T21:21:09.745 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-23T21:21:09.745 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-23T21:21:09.745 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-23T21:21:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:10.098 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":4,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":2685858434},{"type":"v1","addr":"172.21.15.192:6833","nonce":2685858434}]},"active_addr":"172.21.15.192:6833/2685858434","active_change":"2024-04-23T21:21:01.586466+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4112,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":4267745442}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":3079340920}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":2403878506}]}]} 2024-04-23T21:21:10.104 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-23T21:21:10.104 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-23T21:21:10.104 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:21:10.388 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:10.389 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":15,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:21:09.720609+0000","last_up_change":"2024-04-23T21:21:07.708812+0000","last_in_change":"2024-04-23T21:20:59.765723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T21:21:08.380744+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-23T21:21:09.608446+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6817","nonce":1285376165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6819","nonce":1285376165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6823","nonce":1285376165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6821","nonce":1285376165}]},"public_addr":"172.21.15.111:6817/1285376165","cluster_addr":"172.21.15.111:6819/1285376165","heartbeat_back_addr":"172.21.15.111:6823/1285376165","heartbeat_front_addr":"172.21.15.111:6821/1285376165","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6801","nonce":2687705294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6803","nonce":2687705294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6807","nonce":2687705294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6805","nonce":2687705294}]},"public_addr":"172.21.15.192:6801/2687705294","cluster_addr":"172.21.15.192:6803/2687705294","heartbeat_back_addr":"172.21.15.192:6807/2687705294","heartbeat_front_addr":"172.21.15.192:6805/2687705294","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.914033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.903496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.934930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:05.001485+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.979253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.839146+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.963622+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.946000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:21:10.401 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-23T21:21:10.401 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:21:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:10.685 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":15,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:21:09.720609+0000","last_up_change":"2024-04-23T21:21:07.708812+0000","last_in_change":"2024-04-23T21:20:59.765723+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T21:21:08.380744+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-23T21:21:09.608446+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6817","nonce":1285376165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6819","nonce":1285376165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6823","nonce":1285376165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":1285376165},{"type":"v1","addr":"172.21.15.111:6821","nonce":1285376165}]},"public_addr":"172.21.15.111:6817/1285376165","cluster_addr":"172.21.15.111:6819/1285376165","heartbeat_back_addr":"172.21.15.111:6823/1285376165","heartbeat_front_addr":"172.21.15.111:6821/1285376165","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6801","nonce":2687705294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6803","nonce":2687705294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6807","nonce":2687705294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2687705294},{"type":"v1","addr":"172.21.15.192:6805","nonce":2687705294}]},"public_addr":"172.21.15.192:6801/2687705294","cluster_addr":"172.21.15.192:6803/2687705294","heartbeat_back_addr":"172.21.15.192:6807/2687705294","heartbeat_front_addr":"172.21.15.192:6805/2687705294","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.914033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.903496+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.934930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:05.001485+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.979253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.839146+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.963622+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.946000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:21:10.698 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-23T21:21:10.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-23T21:21:10.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-23T21:21:10.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-23T21:21:10.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-23T21:21:10.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-23T21:21:10.700 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-23T21:21:10.701 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-23T21:21:10.946 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T21:21:10.948 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T21:21:10.948 INFO:teuthology.orchestra.run.smithi111.stdout:55834574852 2024-04-23T21:21:10.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T21:21:10.949 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T21:21:10.957 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T21:21:10.961 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T21:21:10.965 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:10.965 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T21:21:11.021 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.021 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T21:21:11.569 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.569 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.585 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2024-04-23T21:21:11.590 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.597 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:11.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2024-04-23T21:21:11.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.7 2024-04-23T21:21:11.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.1 2024-04-23T21:21:11.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-04-23T21:21:11.620 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2024-04-23T21:21:11.636 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:11.647 INFO:teuthology.orchestra.run.smithi111.stdout:55834574849 2024-04-23T21:21:11.649 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.6 2024-04-23T21:21:11.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.0 2024-04-23T21:21:12.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T21:21:12.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T21:21:12.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T21:21:12.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T21:21:12.619 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T21:21:12.634 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T21:21:12.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T21:21:12.661 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T21:21:13.157 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2024-04-23T21:21:13.176 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.197 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.210 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2024-04-23T21:21:13.210 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.233 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.250 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2024-04-23T21:21:13.251 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.7 2024-04-23T21:21:13.264 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.269 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.273 INFO:teuthology.orchestra.run.smithi111.stdout:55834574852 2024-04-23T21:21:13.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2024-04-23T21:21:13.284 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-04-23T21:21:13.286 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.297 INFO:teuthology.orchestra.run.smithi111.stdout:55834574851 2024-04-23T21:21:13.300 INFO:teuthology.orchestra.run.smithi111.stdout:55834574850 2024-04-23T21:21:13.315 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.6 2024-04-23T21:21:13.315 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.0 2024-04-23T21:21:13.316 DEBUG:teuthology.parallel:result is None 2024-04-23T21:21:13.316 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T21:21:13.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:13.638 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:13.639 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:13.650 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T21:21:13.592995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217396,"kb_used_data":1208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757324,"statfs":{"total":773094113280,"available":772871499776,"internally_reserved":0,"allocated":1236992,"data_stored":621454,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000406"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813286+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:10.813286+0000","last_peered":"2024-04-23T21:21:10.813286+0000","last_clean":"2024-04-23T21:21:10.813286+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:10.813286+0000","last_undegraded":"2024-04-23T21:21:10.813286+0000","last_fullsized":"2024-04-23T21:21:10.813286+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813276+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:10.813276+0000","last_peered":"2024-04-23T21:21:10.813276+0000","last_clean":"2024-04-23T21:21:10.813276+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:10.813276+0000","last_undegraded":"2024-04-23T21:21:10.813276+0000","last_fullsized":"2024-04-23T21:21:10.813276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.730262+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:10.730262+0000","last_peered":"2024-04-23T21:21:10.730262+0000","last_clean":"2024-04-23T21:21:10.730262+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:10.730262+0000","last_undegraded":"2024-04-23T21:21:10.730262+0000","last_fullsized":"2024-04-23T21:21:10.730262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813586+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:10.813586+0000","last_peered":"2024-04-23T21:21:10.813586+0000","last_clean":"2024-04-23T21:21:10.813586+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:10.813586+0000","last_undegraded":"2024-04-23T21:21:10.813586+0000","last_fullsized":"2024-04-23T21:21:10.813586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.885301+0000","last_change":"2024-04-23T21:21:10.740377+0000","last_active":"2024-04-23T21:21:10.885301+0000","last_peered":"2024-04-23T21:21:10.885301+0000","last_clean":"2024-04-23T21:21:10.885301+0000","last_became_active":"2024-04-23T21:21:10.740100+0000","last_became_peered":"2024-04-23T21:21:10.740100+0000","last_unstale":"2024-04-23T21:21:10.885301+0000","last_undegraded":"2024-04-23T21:21:10.885301+0000","last_fullsized":"2024-04-23T21:21:10.885301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:40.075195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.735761+0000","last_change":"2024-04-23T21:21:09.734141+0000","last_active":"2024-04-23T21:21:10.735761+0000","last_peered":"2024-04-23T21:21:10.735761+0000","last_clean":"2024-04-23T21:21:10.735761+0000","last_became_active":"2024-04-23T21:21:09.733823+0000","last_became_peered":"2024-04-23T21:21:09.733823+0000","last_unstale":"2024-04-23T21:21:10.735761+0000","last_undegraded":"2024-04-23T21:21:10.735761+0000","last_fullsized":"2024-04-23T21:21:10.735761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:27:29.465371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736858+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:10.736858+0000","last_peered":"2024-04-23T21:21:10.736858+0000","last_clean":"2024-04-23T21:21:10.736858+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:10.736858+0000","last_undegraded":"2024-04-23T21:21:10.736858+0000","last_fullsized":"2024-04-23T21:21:10.736858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813179+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:10.813179+0000","last_peered":"2024-04-23T21:21:10.813179+0000","last_clean":"2024-04-23T21:21:10.813179+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:10.813179+0000","last_undegraded":"2024-04-23T21:21:10.813179+0000","last_fullsized":"2024-04-23T21:21:10.813179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736898+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:10.736898+0000","last_peered":"2024-04-23T21:21:10.736898+0000","last_clean":"2024-04-23T21:21:10.736898+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:10.736898+0000","last_undegraded":"2024-04-23T21:21:10.736898+0000","last_fullsized":"2024-04-23T21:21:10.736898+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":43398,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":309817,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:13.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:13.933 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:13.933 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:13.945 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T21:21:13.592995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217396,"kb_used_data":1208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757324,"statfs":{"total":773094113280,"available":772871499776,"internally_reserved":0,"allocated":1236992,"data_stored":621454,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000406"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813286+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:10.813286+0000","last_peered":"2024-04-23T21:21:10.813286+0000","last_clean":"2024-04-23T21:21:10.813286+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:10.813286+0000","last_undegraded":"2024-04-23T21:21:10.813286+0000","last_fullsized":"2024-04-23T21:21:10.813286+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813276+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:10.813276+0000","last_peered":"2024-04-23T21:21:10.813276+0000","last_clean":"2024-04-23T21:21:10.813276+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:10.813276+0000","last_undegraded":"2024-04-23T21:21:10.813276+0000","last_fullsized":"2024-04-23T21:21:10.813276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.730262+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:10.730262+0000","last_peered":"2024-04-23T21:21:10.730262+0000","last_clean":"2024-04-23T21:21:10.730262+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:10.730262+0000","last_undegraded":"2024-04-23T21:21:10.730262+0000","last_fullsized":"2024-04-23T21:21:10.730262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813586+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:10.813586+0000","last_peered":"2024-04-23T21:21:10.813586+0000","last_clean":"2024-04-23T21:21:10.813586+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:10.813586+0000","last_undegraded":"2024-04-23T21:21:10.813586+0000","last_fullsized":"2024-04-23T21:21:10.813586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.885301+0000","last_change":"2024-04-23T21:21:10.740377+0000","last_active":"2024-04-23T21:21:10.885301+0000","last_peered":"2024-04-23T21:21:10.885301+0000","last_clean":"2024-04-23T21:21:10.885301+0000","last_became_active":"2024-04-23T21:21:10.740100+0000","last_became_peered":"2024-04-23T21:21:10.740100+0000","last_unstale":"2024-04-23T21:21:10.885301+0000","last_undegraded":"2024-04-23T21:21:10.885301+0000","last_fullsized":"2024-04-23T21:21:10.885301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:40.075195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.735761+0000","last_change":"2024-04-23T21:21:09.734141+0000","last_active":"2024-04-23T21:21:10.735761+0000","last_peered":"2024-04-23T21:21:10.735761+0000","last_clean":"2024-04-23T21:21:10.735761+0000","last_became_active":"2024-04-23T21:21:09.733823+0000","last_became_peered":"2024-04-23T21:21:09.733823+0000","last_unstale":"2024-04-23T21:21:10.735761+0000","last_undegraded":"2024-04-23T21:21:10.735761+0000","last_fullsized":"2024-04-23T21:21:10.735761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:27:29.465371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736858+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:10.736858+0000","last_peered":"2024-04-23T21:21:10.736858+0000","last_clean":"2024-04-23T21:21:10.736858+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:10.736858+0000","last_undegraded":"2024-04-23T21:21:10.736858+0000","last_fullsized":"2024-04-23T21:21:10.736858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813179+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:10.813179+0000","last_peered":"2024-04-23T21:21:10.813179+0000","last_clean":"2024-04-23T21:21:10.813179+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:10.813179+0000","last_undegraded":"2024-04-23T21:21:10.813179+0000","last_fullsized":"2024-04-23T21:21:10.813179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736898+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:10.736898+0000","last_peered":"2024-04-23T21:21:10.736898+0000","last_clean":"2024-04-23T21:21:10.736898+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:10.736898+0000","last_undegraded":"2024-04-23T21:21:10.736898+0000","last_fullsized":"2024-04-23T21:21:10.736898+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":43398,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":309817,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:13.946 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T21:21:13.946 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T21:21:13.946 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-23T21:21:13.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-23T21:21:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:14.264 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T21:21:14.277 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-23T21:21:14.277 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-23T21:21:14.290 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-23T21:21:14.290 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8254cb5a247caa8e3d35e534a1fefe2362a0a4b8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}} 2024-04-23T21:21:14.290 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-04-23T21:21:14.290 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-23T21:21:14.593 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-23T21:21:14.593 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T21:21:14.593 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:14.876 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:14.886 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T21:21:13.592995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217396,"kb_used_data":1208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757324,"statfs":{"total":773094113280,"available":772871499776,"internally_reserved":0,"allocated":1236992,"data_stored":621454,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000406"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813286+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:10.813286+0000","last_peered":"2024-04-23T21:21:10.813286+0000","last_clean":"2024-04-23T21:21:10.813286+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:10.813286+0000","last_undegraded":"2024-04-23T21:21:10.813286+0000","last_fullsized":"2024-04-23T21:21:10.813286+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813276+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:10.813276+0000","last_peered":"2024-04-23T21:21:10.813276+0000","last_clean":"2024-04-23T21:21:10.813276+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:10.813276+0000","last_undegraded":"2024-04-23T21:21:10.813276+0000","last_fullsized":"2024-04-23T21:21:10.813276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.730262+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:10.730262+0000","last_peered":"2024-04-23T21:21:10.730262+0000","last_clean":"2024-04-23T21:21:10.730262+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:10.730262+0000","last_undegraded":"2024-04-23T21:21:10.730262+0000","last_fullsized":"2024-04-23T21:21:10.730262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813586+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:10.813586+0000","last_peered":"2024-04-23T21:21:10.813586+0000","last_clean":"2024-04-23T21:21:10.813586+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:10.813586+0000","last_undegraded":"2024-04-23T21:21:10.813586+0000","last_fullsized":"2024-04-23T21:21:10.813586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.885301+0000","last_change":"2024-04-23T21:21:10.740377+0000","last_active":"2024-04-23T21:21:10.885301+0000","last_peered":"2024-04-23T21:21:10.885301+0000","last_clean":"2024-04-23T21:21:10.885301+0000","last_became_active":"2024-04-23T21:21:10.740100+0000","last_became_peered":"2024-04-23T21:21:10.740100+0000","last_unstale":"2024-04-23T21:21:10.885301+0000","last_undegraded":"2024-04-23T21:21:10.885301+0000","last_fullsized":"2024-04-23T21:21:10.885301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:40.075195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.735761+0000","last_change":"2024-04-23T21:21:09.734141+0000","last_active":"2024-04-23T21:21:10.735761+0000","last_peered":"2024-04-23T21:21:10.735761+0000","last_clean":"2024-04-23T21:21:10.735761+0000","last_became_active":"2024-04-23T21:21:09.733823+0000","last_became_peered":"2024-04-23T21:21:09.733823+0000","last_unstale":"2024-04-23T21:21:10.735761+0000","last_undegraded":"2024-04-23T21:21:10.735761+0000","last_fullsized":"2024-04-23T21:21:10.735761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:27:29.465371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736858+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:10.736858+0000","last_peered":"2024-04-23T21:21:10.736858+0000","last_clean":"2024-04-23T21:21:10.736858+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:10.736858+0000","last_undegraded":"2024-04-23T21:21:10.736858+0000","last_fullsized":"2024-04-23T21:21:10.736858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813179+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:10.813179+0000","last_peered":"2024-04-23T21:21:10.813179+0000","last_clean":"2024-04-23T21:21:10.813179+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:10.813179+0000","last_undegraded":"2024-04-23T21:21:10.813179+0000","last_fullsized":"2024-04-23T21:21:10.813179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736898+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:10.736898+0000","last_peered":"2024-04-23T21:21:10.736898+0000","last_clean":"2024-04-23T21:21:10.736898+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:10.736898+0000","last_undegraded":"2024-04-23T21:21:10.736898+0000","last_fullsized":"2024-04-23T21:21:10.736898+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":43398,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":309817,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:14.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:15.169 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:15.169 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:15.181 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-23T21:21:13.592995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217396,"kb_used_data":1208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757324,"statfs":{"total":773094113280,"available":772871499776,"internally_reserved":0,"allocated":1236992,"data_stored":621454,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000406"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813286+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:10.813286+0000","last_peered":"2024-04-23T21:21:10.813286+0000","last_clean":"2024-04-23T21:21:10.813286+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:10.813286+0000","last_undegraded":"2024-04-23T21:21:10.813286+0000","last_fullsized":"2024-04-23T21:21:10.813286+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813276+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:10.813276+0000","last_peered":"2024-04-23T21:21:10.813276+0000","last_clean":"2024-04-23T21:21:10.813276+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:10.813276+0000","last_undegraded":"2024-04-23T21:21:10.813276+0000","last_fullsized":"2024-04-23T21:21:10.813276+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.730262+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:10.730262+0000","last_peered":"2024-04-23T21:21:10.730262+0000","last_clean":"2024-04-23T21:21:10.730262+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:10.730262+0000","last_undegraded":"2024-04-23T21:21:10.730262+0000","last_fullsized":"2024-04-23T21:21:10.730262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813586+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:10.813586+0000","last_peered":"2024-04-23T21:21:10.813586+0000","last_clean":"2024-04-23T21:21:10.813586+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:10.813586+0000","last_undegraded":"2024-04-23T21:21:10.813586+0000","last_fullsized":"2024-04-23T21:21:10.813586+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.885301+0000","last_change":"2024-04-23T21:21:10.740377+0000","last_active":"2024-04-23T21:21:10.885301+0000","last_peered":"2024-04-23T21:21:10.885301+0000","last_clean":"2024-04-23T21:21:10.885301+0000","last_became_active":"2024-04-23T21:21:10.740100+0000","last_became_peered":"2024-04-23T21:21:10.740100+0000","last_unstale":"2024-04-23T21:21:10.885301+0000","last_undegraded":"2024-04-23T21:21:10.885301+0000","last_fullsized":"2024-04-23T21:21:10.885301+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:23:40.075195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.735761+0000","last_change":"2024-04-23T21:21:09.734141+0000","last_active":"2024-04-23T21:21:10.735761+0000","last_peered":"2024-04-23T21:21:10.735761+0000","last_clean":"2024-04-23T21:21:10.735761+0000","last_became_active":"2024-04-23T21:21:09.733823+0000","last_became_peered":"2024-04-23T21:21:09.733823+0000","last_unstale":"2024-04-23T21:21:10.735761+0000","last_undegraded":"2024-04-23T21:21:10.735761+0000","last_fullsized":"2024-04-23T21:21:10.735761+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:27:29.465371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736858+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:10.736858+0000","last_peered":"2024-04-23T21:21:10.736858+0000","last_clean":"2024-04-23T21:21:10.736858+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:10.736858+0000","last_undegraded":"2024-04-23T21:21:10.736858+0000","last_fullsized":"2024-04-23T21:21:10.736858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.813179+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:10.813179+0000","last_peered":"2024-04-23T21:21:10.813179+0000","last_clean":"2024-04-23T21:21:10.813179+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:10.813179+0000","last_undegraded":"2024-04-23T21:21:10.813179+0000","last_fullsized":"2024-04-23T21:21:10.813179+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-23T21:21:10.736898+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:10.736898+0000","last_peered":"2024-04-23T21:21:10.736898+0000","last_clean":"2024-04-23T21:21:10.736898+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:10.736898+0000","last_undegraded":"2024-04-23T21:21:10.736898+0000","last_fullsized":"2024-04-23T21:21:10.736898+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":918560,"data_compressed":43398,"data_compressed_allocated":57344,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":184320,"data_stored":309817,"data_compressed":12159,"data_compressed_allocated":16384,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":459280,"data_compressed":21699,"data_compressed_allocated":28672,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:15.181 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T21:21:15.181 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:epoch 17 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:fsid 3556a001-ee44-4b6e-8d50-1bc45ce74758 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:created 2024-04-23T21:20:57.391123+0000 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:modified 2024-04-23T21:21:11.736770+0000 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:crush_version 4 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:full_ratio 0.95 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:backfillfull_ratio 0.9 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:nearfull_ratio 0.85 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:require_min_compat_client luminous 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:min_compat_client jewel 2024-04-23T21:21:15.464 INFO:teuthology.orchestra.run.smithi111.stdout:require_osd_release squid 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:stretch_mode_enabled false 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 application rbd read_balance_score 2.00 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:max_osd 8 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/1367522096,v1:172.21.15.111:6809/1367522096] [v2:172.21.15.111:6810/1367522096,v1:172.21.15.111:6811/1367522096] exists,up 30c47d3d-583d-4987-9b86-247b80cb80ae 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/1285376165,v1:172.21.15.111:6817/1285376165] [v2:172.21.15.111:6818/1285376165,v1:172.21.15.111:6819/1285376165] exists,up d7c695c5-c361-4c22-b4e7-bd16572f0550 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/698077721,v1:172.21.15.111:6801/698077721] [v2:172.21.15.111:6802/698077721,v1:172.21.15.111:6803/698077721] exists,up 46e4a0e9-b403-4a09-92ca-57245c3fec30 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/4030519478,v1:172.21.15.111:6825/4030519478] [v2:172.21.15.111:6826/4030519478,v1:172.21.15.111:6827/4030519478] exists,up c18de0c4-3fd5-46a0-85a6-f813c8202ea8 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/1513005997,v1:172.21.15.192:6817/1513005997] [v2:172.21.15.192:6818/1513005997,v1:172.21.15.192:6819/1513005997] exists,up d0562ad9-c051-475a-a40b-248d222c8921 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/2687705294,v1:172.21.15.192:6801/2687705294] [v2:172.21.15.192:6802/2687705294,v1:172.21.15.192:6803/2687705294] exists,up 344f8beb-b979-4248-92df-c34cd560be6a 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6823/2824610433,v1:172.21.15.192:6825/2824610433] [v2:172.21.15.192:6826/2824610433,v1:172.21.15.192:6827/2824610433] exists,up 62434311-7dc6-4a3e-bd07-1e8915b37709 2024-04-23T21:21:15.465 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/1007335403,v1:172.21.15.192:6809/1007335403] [v2:172.21.15.192:6810/1007335403,v1:172.21.15.192:6811/1007335403] exists,up db260d92-d3d7-464e-ab34-0bedd2991992 2024-04-23T21:21:15.477 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6808/1367522096,v1:172.21.15.111:6809/1367522096] [v2:172.21.15.111:6810/1367522096,v1:172.21.15.111:6811/1367522096] exists,up 30c47d3d-583d-4987-9b86-247b80cb80ae', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6816/1285376165,v1:172.21.15.111:6817/1285376165] [v2:172.21.15.111:6818/1285376165,v1:172.21.15.111:6819/1285376165] exists,up d7c695c5-c361-4c22-b4e7-bd16572f0550', 'osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6800/698077721,v1:172.21.15.111:6801/698077721] [v2:172.21.15.111:6802/698077721,v1:172.21.15.111:6803/698077721] exists,up 46e4a0e9-b403-4a09-92ca-57245c3fec30', 'osd.3 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.111:6824/4030519478,v1:172.21.15.111:6825/4030519478] [v2:172.21.15.111:6826/4030519478,v1:172.21.15.111:6827/4030519478] exists,up c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'osd.4 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/1513005997,v1:172.21.15.192:6817/1513005997] [v2:172.21.15.192:6818/1513005997,v1:172.21.15.192:6819/1513005997] exists,up d0562ad9-c051-475a-a40b-248d222c8921', 'osd.5 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/2687705294,v1:172.21.15.192:6801/2687705294] [v2:172.21.15.192:6802/2687705294,v1:172.21.15.192:6803/2687705294] exists,up 344f8beb-b979-4248-92df-c34cd560be6a', 'osd.6 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6823/2824610433,v1:172.21.15.192:6825/2824610433] [v2:172.21.15.192:6826/2824610433,v1:172.21.15.192:6827/2824610433] exists,up 62434311-7dc6-4a3e-bd07-1e8915b37709', 'osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/1007335403,v1:172.21.15.192:6809/1007335403] [v2:172.21.15.192:6810/1007335403,v1:172.21.15.192:6811/1007335403] exists,up db260d92-d3d7-464e-ab34-0bedd2991992'] 2024-04-23T21:21:15.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-04-23T21:21:15.575 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "name": "mon.a", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster": "ceph", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "admin_socket_mode": "", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_client_required": "cephx, none", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_cluster_required": "cephx", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_debug": "true", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_service_required": "cephx", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-23T21:21:15.576 INFO:teuthology.orchestra.run.smithi111.stdout: "auth_supported": "cephx", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_aio": "true", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_aio_poll_ms": "250", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_aio_reap_max": "16", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_async_discard_threads": "0", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_block_size": "4096", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_debug_aio": "false", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_debug_inflight_ios": "false", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_enable_discard": "false", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_flock_retry": "3", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_inject_crash": "0", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-23T21:21:15.577 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_ioring": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_ioring_hipri": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bdev_type": "", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_alloc_size": "1048576", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_allocator": "hybrid", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_buffered_io": "true", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_check_for_zeros": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_compact_log_sync": "false", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-23T21:21:15.578 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_max_log_runway": "4194304", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_max_prefetch": "1048576", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_min_flush_size": "524288", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_min_log_runway": "1048576", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_replay_recovery": "false", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluefs_sync_write": "false", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_allocation_from_file": "true", 2024-04-23T21:21:15.579 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_allocator": "hybrid", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_create": "true", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_db_create": "false", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_db_path": "", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_db_size": "0", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_path": "", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_preallocate_file": "false", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_size": "107374182400", 2024-04-23T21:21:15.580 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_wal_create": "false", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_wal_path": "", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_block_wal_size": "100663296", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bluefs": "true", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-23T21:21:15.581 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_autotune": "true", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_size": "0", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cache_type": "2q", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-23T21:21:15.582 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_clone_cow": "true", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_mode": "none", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_csum_type": "crc32c", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_freelist": "false", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-23T21:21:15.583 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-23T21:21:15.584 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_small_allocations": "0", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_default_buffered_read": "true", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_default_buffered_write": "false", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-23T21:21:15.585 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-23T21:21:15.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-23T21:21:15.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-23T21:21:15.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-23T21:21:15.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-23T21:21:15.586 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fail_eio": "false", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-23T21:21:15.587 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-23T21:21:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-23T21:21:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_mount": "false", 2024-04-23T21:21:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-23T21:21:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_umount": "false", 2024-04-23T21:21:15.588 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-23T21:21:15.589 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_ignore_data_csum": "false", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-23T21:21:15.590 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_log_op_age": "5.000000", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_alloc_size": "0", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_blob_size": "0", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-23T21:21:15.591 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-23T21:21:15.592 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_max_deferred_txc": "32", 2024-04-23T21:21:15.592 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_min_alloc_size": "0", 2024-04-23T21:21:15.592 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_nid_prealloc": "1024", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-23T21:21:15.593 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_retry_disk_reads": "3", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_rocksdb_cf": "true", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-23T21:21:15.594 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_spdk_mem": "512", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-23T21:21:15.595 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_tracing": "false", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-23T21:21:15.596 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "bluestore_zero_block_detection": "false", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-23T21:21:15.597 INFO:teuthology.orchestra.run.smithi111.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_cluster_require_signatures": "false", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_cluster_require_version": "2", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_require_signatures": "false", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_require_version": "2", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_service_require_signatures": "false", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_service_require_version": "2", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "cephx_sign_messages": "true", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "chdir": "", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_acl_type": "", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_asio_thread_count": "2", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_cache_mid": "0.750000", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_cache_size": "16384", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_caps_release_delay": "5", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_check_pool_perm": "true", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-23T21:21:15.598 INFO:teuthology.orchestra.run.smithi111.stdout: "client_debug_force_sync_read": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_debug_getattr_caps": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_debug_inject_tick_delay": "0", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_die_on_failed_remount": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_dirsize_rbytes": "true", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_force_lazyio": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_fs": "", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_inject_release_failure": "false", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_max_inline_size": "4096", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_mds_namespace": "", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_metadata": "", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_mount_gid": "-1", 2024-04-23T21:21:15.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_mount_timeout": "300", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_mount_uid": "-1", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_mountpoint": "/", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_notify_timeout": "10", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc": "true", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc_max_dirty": "104857600", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc_max_objects": "1000", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc_size": "209715200", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_oc_target_dirty": "8388608", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_permissions": "true", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_quota": "true", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_quota_df": "true", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_readahead_max_bytes": "0", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_readahead_max_periods": "4", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_readahead_min": "131072", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_reconnect_stale": "false", 2024-04-23T21:21:15.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_shutdown_timeout": "30", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_snapdir": ".snap", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_tick_interval": "1", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_trace": "", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_try_dentry_invalidate": "false", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_use_faked_inos": "false", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_use_random_mds": "false", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_graylog": "false", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_graylog_port": "12201", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_monitors": "default=true", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_syslog": "false", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "clog_to_syslog_level": "info", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "-", 2024-04-23T21:21:15.601 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_network": "", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_network_interface": "", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "compressor_zlib_isal": "false", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "compressor_zlib_level": "5", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "compressor_zlib_winsize": "-15", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "compressor_zstd_level": "1", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_alien_op_num_threads": "6", 2024-04-23T21:21:15.602 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_osd_stat_interval": "0", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crimson_seastar_cpu_cores": "", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_location": "", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_location_hook": "", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_location_hook_timeout": "10", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "daemonize": "false", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "daos_pool": "tank", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_alienstore": "0/5", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_allow_any_pool_priority": "false", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_asok": "1/5", 2024-04-23T21:21:15.603 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_asok_assert_abort": "false", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_asserts_on_shutdown": "true", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_auth": "20/20", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_bdev": "1/3", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_bluefs": "1/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_bluestore": "1/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_buffer": "0/1", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_ceph_exporter": "1/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_cephfs_mirror": "0/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_cephsqlite": "0/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_client": "0/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_compressor": "1/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_context": "0/1", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_crimson_interrupt": "0/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_crush": "1/1", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_crypto": "1/5", 2024-04-23T21:21:15.604 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_cyanstore": "0/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_deliberately_leak_memory": "false", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_disable_randomized_ping": "false", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_dpdk": "1/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_eventtrace": "1/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_filer": "0/1", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_filestore": "1/3", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_finisher": "1/1", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_fuse": "1/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_heartbeat_testing_span": "0", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_heartbeatmap": "1/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_javaclient": "1/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_journal": "1/3", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_journaler": "0/5", 2024-04-23T21:21:15.605 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_kstore": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_lockdep": "0/1", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mclock": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_balancer": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_locker": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_log": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_log_expire": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_migrator": "1/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mds_quiesce": "3/5", 2024-04-23T21:21:15.606 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_memstore": "1/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mgr": "2/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mgrc": "1/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_mon": "20/20", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_monc": "0/10", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_ms": "1/1", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_none": "0/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_objclass": "0/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_objectcacher": "0/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_objecter": "0/1", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_optracker": "0/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_osd": "1/5", 2024-04-23T21:21:15.607 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_paxos": "20/20", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_perfcounter": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_prioritycache": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rados": "0/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rbd": "0/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rbd_mirror": "0/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rbd_pwl": "0/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rbd_replay": "0/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_refs": "0/0", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_reserver": "1/1", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_access": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_datacache": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_dbstore": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_flight": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_notification": "1/5", 2024-04-23T21:21:15.608 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rgw_sync": "1/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_rocksdb": "4/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_backref": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_cache": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_cleaner": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_device": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_epm": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_journal": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_lba": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_odata": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_omap": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_onode": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_t": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_seastore_tm": "0/5", 2024-04-23T21:21:15.609 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_striper": "0/1", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_test": "0/5", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_throttle": "1/1", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_timer": "0/1", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_tp": "0/5", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "debug_trace": "1/5", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "device_failure_prediction_mode": "none", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "err_to_graylog": "false", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "err_to_journald": "false", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "err_to_stderr": "true", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "err_to_syslog": "false", 2024-04-23T21:21:15.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event_tracing": "false", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_addr": "0.0.0.0", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_http_port": "9926", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_prio_limit": "5", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_sort_metrics": "true", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "exporter_stats_period": "5", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "fake_statfs_for_testing": "0", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "fatal_signal_handlers": "true", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filer_max_purge_ops": "10", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filer_max_truncate_ops": "128", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_apply_finisher_threads": "1", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_blackhole": "false", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_btrfs_clone_range": "true", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_btrfs_snap": "true", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_caller_concurrency": "10", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_collect_device_partition_information": "true", 2024-04-23T21:21:15.611 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_commit_timeout": "600.000000", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_debug_inject_read_err": "false", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_debug_omap_check": "false", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_debug_verify_split": "false", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_dump_file": "", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fadvise": "true", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fail_eio": "true", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fd_cache_shards": "16", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fd_cache_size": "128", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fiemap": "false", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fiemap_threshold": "4096", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-23T21:21:15.612 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_inject_stall": "0", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_journal_parallel": "false", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_journal_trailing": "false", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_journal_writeahead": "false", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_kill_at": "0", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattrs": "0", 2024-04-23T21:21:15.613 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_xattr_value_size": "0", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_merge_threshold": "-10", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_odsync_write": "false", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_omap_backend": "rocksdb", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_omap_backend_path": "", 2024-04-23T21:21:15.614 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_op_thread_timeout": "60", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_op_threads": "2", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_punch_hole": "false", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_queue_max_ops": "50", 2024-04-23T21:21:15.615 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_seek_data_hole": "false", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_sloppy_crc": "false", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_splice": "false", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_split_multiple": "2", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_split_rand_factor": "20", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_update_to": "1000", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_enable": "true", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-23T21:21:15.616 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_xfs_extsize": "false", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "filestore_zfs_snap": "false", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fio_dir": "/tmp/fio", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_allow_other": "true", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_atomic_o_trunc": "true", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_big_writes": "true", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_debug": "false", 2024-04-23T21:21:15.617 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_default_permissions": "false", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_disable_pagecache": "false", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_max_write": "0", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_multithreaded": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_require_active_mds": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_set_user_groups": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_splice_move": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_splice_read": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_splice_write": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "fuse_use_invalidate_cb": "true", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "gss_target_name": "ceph", 2024-04-23T21:21:15.618 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_file": "", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_inject_failure": "0", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_interval": "5", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "host": "smithi111", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-23T21:21:15.619 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "inject_early_sigterm": "false", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "jaeger_agent_port": "6799", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "jaeger_tracing_enable": "false", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_aio": "true", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_align_min_size": "65536", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_block_align": "true", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_block_size": "4096", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_dio": "true", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_discard": "false", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_force_aio": "false", 2024-04-23T21:21:15.620 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_ignore_corruption": "false", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_max_write_bytes": "10485760", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_max_write_entries": "100", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_replay_from": "0", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_write_header_frequency": "0", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journal_zero_on_create": "false", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journaler_prefetch_periods": "10", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journaler_prezero_periods": "5", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "journaler_write_head_interval": "15", 2024-04-23T21:21:15.621 INFO:teuthology.orchestra.run.smithi111.stdout: "key": "", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "keyfile": "", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_backend": "rocksdb", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_default_stripe_size": "65536", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_fsck_on_mount": "false", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_max_bytes": "67108864", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_max_ops": "512", 2024-04-23T21:21:15.622 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_nid_prealloc": "1024", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_onode_map_size": "1024", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_sync_submit_transaction": "false", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "kstore_sync_transaction": "false", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "librados_thread_count": "2", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "lockdep": "false", 2024-04-23T21:21:15.623 INFO:teuthology.orchestra.run.smithi111.stdout: "lockdep_force_backtrace": "false", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_coarse_timestamps": "true", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_flush_on_exit": "false", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_graylog_host": "127.0.0.1", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_graylog_port": "12201", 2024-04-23T21:21:15.624 INFO:teuthology.orchestra.run.smithi111.stdout: "log_max_new": "1000", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_max_recent": "10000", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_stderr_prefix": "", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_stop_at_utilization": "0.970000", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_file": "true", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_graylog": "false", 2024-04-23T21:21:15.625 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_journald": "false", 2024-04-23T21:21:15.626 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_stderr": "false", 2024-04-23T21:21:15.626 INFO:teuthology.orchestra.run.smithi111.stdout: "log_to_syslog": "false", 2024-04-23T21:21:15.626 INFO:teuthology.orchestra.run.smithi111.stdout: "max_rotating_auth_attempts": "10", 2024-04-23T21:21:15.626 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-23T21:21:15.627 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_action_on_write_error": "1", 2024-04-23T21:21:15.627 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_alternate_name_max": "8192", 2024-04-23T21:21:15.627 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_asio_thread_count": "2", 2024-04-23T21:21:15.627 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_export_pin": "true", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_fragment_dirs": "true", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_fragment_interval": "5", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-23T21:21:15.628 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_interval": "10", 2024-04-23T21:21:15.629 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_max": "-1", 2024-04-23T21:21:15.629 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_max_until": "-1", 2024-04-23T21:21:15.629 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_merge_size": "50", 2024-04-23T21:21:15.629 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_midchunk": "0.300000", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_min_start": "0.200000", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_minchunk": "0.001000", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_mode": "0", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_need_max": "1.200000", 2024-04-23T21:21:15.630 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_need_min": "0.800000", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_overload_epochs": "2", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_split_bits": "3", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-23T21:21:15.631 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_split_size": "10000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_target_decay": "10.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_beacon_grace": "15.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_beacon_interval": "4.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_mid": "0.700000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_quiesce_delay": "0", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-23T21:21:15.632 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_release_free_interval": "10", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_reservation": "0.050000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_trim_interval": "1", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cache_trim_threshold": "262144", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_client_prealloc_inos": "1000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_connect_bootstrapping": "false", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_damage_table_max_entries": "10000", 2024-04-23T21:21:15.633 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_debug_auth_pins": "false", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_debug_frag": "false", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_debug_scatterstat": "false", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_debug_subtrees": "false", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_decay_halflife": "5.000000", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_default_dir_hash": "2", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_defer_session_stale": "true", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_deny_all_reconnect": "false", 2024-04-23T21:21:15.634 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dir_keys_per_op": "16384", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dir_max_commit_size": "10", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dir_max_entries": "0", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dir_prefetch": "true", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dump_cache_on_map": "false", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_early_reply": "true", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_enable_op_tracker": "true", 2024-04-23T21:21:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_enforce_unique_name": "true", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_export_ephemeral_random": "true", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_health_summarize_threshold": "10", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-23T21:21:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_health_dummy": "false", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_migrator_session_race": "false", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_join_fs": "", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_journal_format": "1", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_create_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_export_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_import_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_journal_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_journal_expire_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_journal_replay_at": "0", 2024-04-23T21:21:15.637 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_link_at": "0", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_mdstable_at": "0", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_openc_at": "0", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_rename_at": "0", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_shutdown_at": "0", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_event_large_threshold": "524288", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_events_per_segment": "1024", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-23T21:21:15.638 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_max_events": "-1", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_max_segments": "128", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_pause": "false", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_segment_size": "0", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_trim_threshold": "128", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_log_warn_factor": "2.000000", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_caps_per_client": "1048576", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_completed_flushes": "100000", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_completed_requests": "100000", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_export_size": "20971520", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_file_recover": "32", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_purge_files": "64", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_purge_ops": "8192", 2024-04-23T21:21:15.639 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_max_snaps_per_dir": "100", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_metrics_update_interval": "2", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_min_caps_per_client": "100", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_min_caps_working_set": "10000", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_numa_node": "-1", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_complaint_time": "30.000000", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_history_duration": "600", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_history_size": "20", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_history_slow_op_size": "20", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-23T21:21:15.640 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_op_log_threshold": "5", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_ping_grace": "15", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_ping_interval": "5", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_max_caps": "30000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_recall_warning_threshold": "262144", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_replay_interval": "1.000000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_root_ino_gid": "0", 2024-04-23T21:21:15.641 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_root_ino_uid": "0", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_shutdown_check": "0", 2024-04-23T21:21:15.642 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_skip_ino": "0", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_snap_max_uid": "4294967294", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_snap_min_uid": "0", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_snap_rstat": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_standby_replay_damaged": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_symlink_recovery": "true", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_thrash_exports": "0", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_thrash_fragments": "0", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_tick_interval": "5.000000", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_valgrind_exit": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_verify_backtrace": "1", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_verify_scatter": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mds_wipe_sessions": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "mempool_debug": "false", 2024-04-23T21:21:15.643 INFO:teuthology.orchestra.run.smithi111.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "memstore_device_bytes": "1073741824", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "memstore_page_set": "false", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "memstore_page_size": "65536", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_client_bytes": "134217728", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_client_messages": "512", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_disabled_modules": "", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-23T21:21:15.644 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_max_pg_creating": "1024", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_max_pg_num_change": "128", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_mds_bytes": "134217728", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_mds_messages": "128", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_mon_bytes": "134217728", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_mon_messages": "128", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_osd_bytes": "536870912", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_osd_messages": "8192", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_pool": "true", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_standby_modules": "true", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_stats_period": "5", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_stats_threshold": "5", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_tick_period": "2", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-23T21:21:15.645 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_allow_pool_delete": "true", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_allow_pool_size_one": "true", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_auth_validate_all_caps": "true", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_bytes": "104857600", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_directed_command_retry": "5", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_hunt_parallel": "3", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_log_interval": "1.000000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-23T21:21:15.646 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_ping_interval": "10.000000", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_client_target_rank": "-1", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_level": "debug", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_file": "true", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_journald": "false", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_stderr": "false", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-23T21:21:15.647 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_compact_on_bootstrap": "false", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_compact_on_start": "false", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_compact_on_trim": "true", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_cpu_threads": "4", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_crush_min_required_version": "hammer", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_daemon_bytes": "419430400", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_data_avail_crit": "5", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_data_avail_warn": "5", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_data_size_warn": "16106127360", 2024-04-23T21:21:15.648 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_dump_json": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_dump_transactions": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_extra_checks": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_no_require_reef": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_no_require_squid": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-23T21:21:15.649 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_down_mkfs_grace": "120", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_down_uptime_grace": "60", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_election_default_strategy": "1", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_election_timeout": "5.000000", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_elector_ping_divisor": "2", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_enable_op_tracker": "true", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_fake_pool_delete": "false", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_force_quorum_join": "false", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_globalid_prealloc": "10000", 2024-04-23T21:21:15.650 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_detail_to_clog": "false", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_log_update_period": "5", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_max_detail": "50", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_to_clog": "true", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_to_clog_interval": "600", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_host": "172.21.15.111,[v2:172.21.15.111:3301,v1:172.21.15.111:6790],172.21.15.192", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_host_override": "", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_initial_members": "", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_lease": "5.000000", 2024-04-23T21:21:15.651 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_log_full_interval": "50", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_log_max": "10000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_log_max_summary": "50", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_log_epochs": "500", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_osd": "10000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_pg_per_osd": "10000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_pool_pg_num": "65536", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mds_force_trim_to": "0", 2024-04-23T21:21:15.652 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mds_skip_sanity": "false", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_memory_autotune": "true", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_memory_target": "2147483648", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_beacon_grace": "30", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_digest_period": "5", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_inactive_grace": "60", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_min_osdmap_epochs": "500", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_complaint_time": "30", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_history_duration": "600", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_history_size": "20", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_history_slow_op_size": "20", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-23T21:21:15.653 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_op_log_threshold": "5", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_auto_mark_in": "false", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_cache_size": "500", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_down_out_interval": "600", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-23T21:21:15.654 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_force_trim_to": "0", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_min_down_reporters": "2", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-23T21:21:15.655 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_report_timeout": "900", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osdmap_full_prune_interval": "10", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osdmap_full_prune_min": "10000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_stuck_threshold": "60", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-23T21:21:15.656 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_probe_timeout": "2.000000", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_reweight_max_change": "0.050000", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_reweight_max_osds": "4", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_scrub_interval": "86400", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_scrub_max_keys": "100", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_scrub_timeout": "300", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_session_timeout": "300", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_smart_report_timeout": "5", 2024-04-23T21:21:15.657 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_stat_smooth_intervals": "6", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_stretch_pool_min_size": "2", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_stretch_pool_size": "4", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_debug": "false", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_provider_kill_at": "0", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_requester_kill_at": "0", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_sync_timeout": "60.000000", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_target_pg_per_osd": "100", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_tick_interval": "5", 2024-04-23T21:21:15.658 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_timecheck_interval": "300.000000", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_use_min_delay_socket": "false", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_older_version_delay": "604800", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_filestore_osds": "false", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-23T21:21:15.659 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_misplaced": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_older_version": "true", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-23T21:21:15.660 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "monmap": "", 2024-04-23T21:21:15.661 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-23T21:21:15.662 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-23T21:21:15.662 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-23T21:21:15.662 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-23T21:21:15.662 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-23T21:21:15.662 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "motr_tracing_enabled": "false", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_op_threads": "3", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_cm": "false", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_device_name": "", 2024-04-23T21:21:15.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_dscp": "96", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_local_gid": "", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_port_num": "1", 2024-04-23T21:21:15.664 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_sl": "3", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_support_srq": "true", 2024-04-23T21:21:15.665 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_rdma_type": "ib", 2024-04-23T21:21:15.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_async_reap_threshold": "5", 2024-04-23T21:21:15.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_before_connect": "false", 2024-04-23T21:21:15.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_ipv4": "true", 2024-04-23T21:21:15.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_ipv6": "false", 2024-04-23T21:21:15.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_msgr1": "true", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_msgr2": "true", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_port_max": "7568", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_port_min": "6800", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_retry_count": "3", 2024-04-23T21:21:15.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_bind_retry_delay": "5", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_blackhole_client": "false", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_blackhole_mds": "false", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_blackhole_mgr": "false", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_blackhole_mon": "false", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_blackhole_osd": "false", 2024-04-23T21:21:15.668 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_client_mode": "crc secure", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_cluster_mode": "crc secure", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_cluster_type": "", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_compress_secure": "false", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_connection_idle_timeout": "900", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_connection_ready_timeout": "10", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_crc_data": "true", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_crc_header": "true", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_die_on_bad_msg": "false", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_die_on_bug": "true", 2024-04-23T21:21:15.669 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_die_on_old_message": "true", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_die_on_skipped_message": "false", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_coremask": "0xF", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_enable_tso": "true", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-23T21:21:15.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_hugepages": "", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_lro": "true", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_memory_channel": "4", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_pmd": "", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_port_id": "0", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_dump_on_send": "false", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_initial_backoff": "0.200000", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_delay_max": "1.000000", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-23T21:21:15.671 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_delay_type": "", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_network_congestion": "0", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_inject_socket_failures": "5000", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_learn_addr_from_peer": "true", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_max_accept_failures": "4", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_max_backoff": "15.000000", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_mon_client_mode": "secure crc", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_mon_service_mode": "secure crc", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_osd_compress_min_size": "1024", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_osd_compress_mode": "none", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-23T21:21:15.672 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_pq_min_cost": "65536", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_public_type": "", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_service_mode": "crc secure", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_tcp_listen_backlog": "512", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_tcp_nodelay": "true", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_tcp_rcvbuf": "0", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "ms_type": "async+posix", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "no_config_file": "false", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_completion_locks_per_session": "32", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_inflight_ops": "1024", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-23T21:21:15.673 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_tick_interval": "5.000000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "objecter_timeout": "10.000000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "objectstore_blackhole": "false", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "openssl_engine_opts": "", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_delay_time": "5.000000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_hist_halflife": "1000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_max_low_ops": "2", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_max_ops": "4", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_agent_slop": "0.020000", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-23T21:21:15.674 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_asio_thread_count": "2", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_async_recovery_min_cost": "100", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backfill_scan_max": "512", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backfill_scan_min": "64", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backoff_on_degraded": "false", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backoff_on_peering": "false", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_backoff_on_unfound": "true", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_beacon_report_interval": "300", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bench_duration": "30", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bench_max_block_size": "67108864", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_blkin_trace_all": "false", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-23T21:21:15.675 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_check_for_log_corruption": "false", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_class_update_on_start": "true", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_client_message_cap": "256", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_client_message_size_cap": "524288000", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_client_op_priority": "63", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_client_watch_timeout": "30", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_command_max_records": "256", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_compact_on_start": "false", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-23T21:21:15.676 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_crush_chooseleaf_type": "0", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_crush_update_on_start": "true", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_crush_update_weight_set": "true", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_feed_pullee": "-1", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-23T21:21:15.677 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_misdirected_ops": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_no_acting_change": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_no_purge_strays": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_op_order": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_shutdown": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-23T21:21:15.678 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_trim_objects": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_verify_snaps": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_keys": "1024", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_stride": "524288", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_default_data_pool_replay_window": "5", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_default_notify_timeout": "30", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_delete_sleep": "0.000000", 2024-04-23T21:21:15.679 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_discard_disconnected_ops": "true", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_enable_op_tracker": "true", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_fast_info": "true", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_fast_shutdown": "true", 2024-04-23T21:21:15.680 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_function_tracing": "false", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_grace": "20", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_interval": "6", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_min_peers": "10", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_min_size": "2000", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_stale": "600", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_hit_set_max_size": "100000", 2024-04-23T21:21:15.681 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_hit_set_min_size": "1000", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_journal_size": "5120", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_kill_backfill_at": "0", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_map_cache_size": "50", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_map_dedup": "true", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_map_message_max": "40", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-23T21:21:15.682 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_map_share_max_epochs": "40", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_attr_name_len": "100", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_attr_size": "0", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_backfills": "1", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_markdown_count": "5", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_markdown_period": "600", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_object_name_len": "2048", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_object_namespace_len": "256", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_object_size": "134217728", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_pg_blocked_by": "16", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_pg_log_entries": "10000", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_pgls": "1024", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_push_cost": "8388608", 2024-04-23T21:21:15.683 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_push_objects": "10", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_scrubs": "3", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_trimming_pgs": "2", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_write_op_reply_len": "64", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_max_write_size": "90", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-23T21:21:15.684 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_profile": "balanced", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-23T21:21:15.685 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_base": "805306368", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_cache_min": "134217728", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_target": "4294967296", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_target_autotune": "false", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_min_pg_log_entries": "250", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_min_recovery_priority": "0", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-23T21:21:15.686 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mon_report_interval": "5", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_num_cache_shards": "32", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_num_op_tracker_shard": "32", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_numa_auto_affinity": "true", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_numa_node": "-1", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_numa_prefer_iface": "true", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_objecter_finishers": "1", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_objectstore": "bluestore", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_objectstore_fuse": "false", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_objectstore_tracing": "false", 2024-04-23T21:21:15.687 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_complaint_time": "30.000000", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_history_duration": "600", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_history_size": "20", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_history_slow_op_size": "20", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_log_threshold": "5", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_shards": "0", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_shards_hdd": "5", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_shards_ssd": "8", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_pq_min_cost": "65536", 2024-04-23T21:21:15.688 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_queue_cut_off": "high", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_op_thread_timeout": "15", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_open_classes_on_start": "true", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_os_flags": "0", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_peering_op_priority": "255", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_delete_cost": "1048576", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_delete_priority": "5", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_log_trim_max": "10000", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_log_trim_min": "100", 2024-04-23T21:21:15.689 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_crimson": "false", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-23T21:21:15.690 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_flags": "0", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_min_size": "0", 2024-04-23T21:21:15.691 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_pg_num": "32", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_pgp_num": "0", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_read_ratio": "70", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_size": "2", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_default_type": "replicated", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_push_per_object_cost": "1000", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recover_clone_overlap": "true", 2024-04-23T21:21:15.692 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_cost": "20971520", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_active": "0", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_max_single_start": "1", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_op_priority": "3", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-23T21:21:15.693 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_priority": "5", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_sleep": "0.000000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_repair_during_recovery": "false", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_requested_scrub_priority": "120", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_auto_repair": "false", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_begin_hour": "0", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_begin_week_day": "0", 2024-04-23T21:21:15.694 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_chunk_max": "25", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_chunk_min": "5", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_cost": "52428800", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_during_recovery": "false", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_end_hour": "0", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_end_week_day": "0", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_event_cost": "4096", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_invalid_stats": "true", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_max_preemptions": "5", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_priority": "5", 2024-04-23T21:21:15.695 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_scrub_sleep": "0.000000", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_skip_data_digest": "false", 2024-04-23T21:21:15.696 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_smart_report_timeout": "5", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_cost": "1048576", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_priority": "5", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-23T21:21:15.697 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-23T21:21:15.698 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-23T21:21:15.698 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-23T21:21:15.698 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-23T21:21:15.698 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_target_transaction_size": "30", 2024-04-23T21:21:15.698 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-23T21:21:15.699 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_tracing": "false", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_use_stale_snap": "false", 2024-04-23T21:21:15.700 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "osdc_blkin_trace_all": "false", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_kill_at": "0", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_max_join_drift": "10", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_min": "500", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_min_wait": "0.050000", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_propose_interval": "1.000000", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_service_trim_max": "500", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_service_trim_min": "250", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_stash_full_interval": "25", 2024-04-23T21:21:15.701 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_trim_max": "500", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "paxos_trim_min": "250", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "perf": "true", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "-", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrv": "", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "public_bind_addr": "-", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "public_network": "", 2024-04-23T21:21:15.702 INFO:teuthology.orchestra.run.smithi111.stdout: "public_network_interface": "", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "qat_compressor_enabled": "false", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "qat_compressor_session_max_number": "256", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rados_mon_op_timeout": "0", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rados_osd_op_timeout": "0", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rados_tracing": "false", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_atime_update_interval": "60", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_balance_parent_reads": "false", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_balance_snap_reads": "false", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_blkin_trace_all": "false", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache": "true", 2024-04-23T21:21:15.703 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_policy": "writearound", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_size": "33554432", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_clone_copy_on_read": "false", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_compression_hint": "none", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_concurrent_management_ops": "10", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_clone_format": "auto", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_data_pool": "", 2024-04-23T21:21:15.704 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_features": "61", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_format": "2", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_map_options": "", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_order": "22", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_pool": "rbd", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_stripe_count": "0", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_default_stripe_unit": "0", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_enable_alloc_hint": "true", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_io_scheduler": "simple", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-23T21:21:15.705 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_order": "24", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_pool": "", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_journal_splay_width": "4", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_localize_parent_reads": "false", 2024-04-23T21:21:15.706 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_localize_snap_reads": "false", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-23T21:21:15.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-23T21:21:15.708 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_mtime_update_interval": "60", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_non_blocking_aio": "true", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_op_thread_timeout": "60", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_op_threads": "1", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_parent_cache_enabled": "false", 2024-04-23T21:21:15.709 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_plugins": "", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_bps_burst": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_bps_limit": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_exclude_ops": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_iops_burst": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_iops_limit": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-23T21:21:15.710 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_read_from_replica_policy": "default", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_skip_partial_discard": "true", 2024-04-23T21:21:15.711 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_tracing": "false", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_validate_names": "true", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd_validate_pool": "true", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "restapi_base_url": "", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "restapi_log_level": "", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_acl_grants_max_num": "100", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_admin_entry": "admin", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_backend_store": "rados", 2024-04-23T21:21:15.712 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_barbican_url": "", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_beast_enable_async": "true", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_counters_cache": "false", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_cache_enabled": "true", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_cache_expiry_interval": "900", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_cache_lru_size": "10000", 2024-04-23T21:21:15.713 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_config_store": "rados", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_content_length_compat": "false", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_copy_obj_progress": "true", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_cors_rules_max_num": "100", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_cross_domain_policy": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_addr": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-23T21:21:15.714 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_password": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_kmip_username": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_require_ssl": "true", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-23T21:21:15.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_addr": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_auth": "token", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_namespace": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_prefix": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_token_file": "", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_curl_buffersize": "524288", 2024-04-23T21:21:15.716 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_curl_low_speed_time": "300", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-23T21:21:15.717 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_d4n_port": "6379", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_log_changes_size": "1000", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_log_num_shards": "128", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_log_window": "30", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-23T21:21:15.718 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-23T21:21:15.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dns_name": "", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dns_s3website_name": "", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_dynamic_resharding": "true", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_gc_threads": "true", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_lc_threads": "true", 2024-04-23T21:21:15.720 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_ops_log": "false", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_quota_threads": "true", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_static_website": "false", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enable_usage_log": "false", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_enforce_swift_acls": "true", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_exit_timeout_secs": "120", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_expose_bucket": "false", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_extended_http_attrs": "", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_filter": "none", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_frontends": "beast port=7480", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_deferred": "50", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_objs": "32", 2024-04-23T21:21:15.721 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_gc_processor_period": "3600", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-23T21:21:15.722 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_init_timeout": "300", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_kafka_connection_idle": "300", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-23T21:21:15.723 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_domain": "", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_password": "", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_password_path": "", 2024-04-23T21:21:15.724 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_project": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_tenant": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_token": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_token_path": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_admin_user": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_api_version": "2", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_barbican_domain": "", 2024-04-23T21:21:15.725 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_barbican_password": "", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_barbican_project": "", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_barbican_user": "", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-23T21:21:15.726 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_url": "", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_debug_interval": "-1", 2024-04-23T21:21:15.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_lock_max_time": "90", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_max_objs": "32", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_max_rules": "1000", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_max_worker": "3", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lc_thread_delay": "0", 2024-04-23T21:21:15.728 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_dnattr": "uid", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_searchfilter": "", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ldap_uri": "", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-23T21:21:15.729 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_log_http_headers": "", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_log_object_name_utc": "false", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_attr_name_len": "0", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_attr_size": "0", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_chunk_size": "4194304", 2024-04-23T21:21:15.730 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_listing_results": "1000", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_notify_retries": "10", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_put_param_size": "1048576", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_put_size": "5368709120", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_max_slo_entries": "1000", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_md_log_max_shards": "64", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_mp_lock_max_time": "600", 2024-04-23T21:21:15.731 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_max_gc": "300", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-23T21:21:15.732 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_num_async_rados_threads": "32", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_num_control_oids": "8", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_numa_node": "-1", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_objexp_chunk_size": "100", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_objexp_gc_interval": "600", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_op_thread_timeout": "600", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_op_tracing": "false", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_opa_token": "", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_opa_url": "", 2024-04-23T21:21:15.733 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_opa_verify_ssl": "true", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ops_log_rados": "false", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_ops_log_socket_path": "", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_period_push_interval": "2.000000", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_cache_lanes": "3", 2024-04-23T21:21:15.734 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_cache_partitions": "3", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_print_continue": "true", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_rados_tracing": "false", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_realm": "", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_realm_id": "", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-23T21:21:15.735 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_region": "", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_request_uri": "", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_reshard_batch_size": "64", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_reshard_max_aio": "128", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_reshard_num_logs": "16", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_reshard_thread_interval": "600", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_resolve_cname": "false", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_run_sync_thread": "true", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-23T21:21:15.736 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_scheduler_type": "throttler", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_script_uri": "", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_service_provider_name": "", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_client_id": "", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_client_secret": "", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_entry": "sts", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_key": "sts", 2024-04-23T21:21:15.737 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_min_session_duration": "900", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sts_token_introspection_url": "", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_account_in_url": "false", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_auth_entry": "auth", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_auth_url": "", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_custom_header": "", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_need_stats": "true", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_tenant_name": "", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_token_expiration": "86400", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_url": "", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_url_prefix": "swift", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-23T21:21:15.738 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_lease_period": "120", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_thread_pool_size": "512", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-23T21:21:15.739 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_comment": "", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_createby": "", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_encoding": "", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_flag": "false", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_origin": "", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_torrent_tracker": "", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_trust_forwarded_https": "false", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_usage_max_shards": "32", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_usage_max_user_shards": "1", 2024-04-23T21:21:15.740 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_use_opa_authz": "false", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_counters_cache": "false", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_max_buckets": "1000", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_policies_max_num": "100", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_user_unique_email": "true", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_verify_ssl": "true", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zone": "", 2024-04-23T21:21:15.741 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zone_id": "", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zonegroup": "", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zonegroup_id": "", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgwlc_auto_session_clear": "true", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_block_size": "4096", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_size": "536870912", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-23T21:21:15.742 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_index_type": "binary_search", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_partition_filters": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_perf": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-23T21:21:15.743 INFO:teuthology.orchestra.run.smithi111.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "run_dir": "/var/run/ceph", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_block_create": "true", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_cache_lru_size": "67108864", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_default_max_object_size": "16777216", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_device_size": "53687091200", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_journal_batch_capacity": "16", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_main_device_type": "SSD", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-23T21:21:15.744 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "seastore_segment_size": "67108864", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "set_keepcaps": "false", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "setgroup": "", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "setuser": "", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "setuser_match_path": "", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "thp": "false", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "threadpool_default_timeout": "60", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout: "throttler_perf_counter": "true" 2024-04-23T21:21:15.745 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:15.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-23T21:21:15.857 INFO:teuthology.orchestra.run.smithi111.stdout:mon.a: {} 2024-04-23T21:21:15.857 INFO:teuthology.orchestra.run.smithi111.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:21:15.868 INFO:teuthology.orchestra.run.smithi111.stdout:mon.b: {} 2024-04-23T21:21:15.868 INFO:teuthology.orchestra.run.smithi111.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:21:15.876 INFO:teuthology.orchestra.run.smithi111.stdout:mon.c: {} 2024-04-23T21:21:15.876 INFO:teuthology.orchestra.run.smithi111.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:21:15.887 DEBUG:teuthology.orchestra.run.smithi192:> type ceph-objectstore-tool 2024-04-23T21:21:15.904 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-23T21:21:15.905 DEBUG:teuthology.orchestra.run.smithi111:> type ceph-objectstore-tool 2024-04-23T21:21:15.919 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-23T21:21:15.919 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-04-23T21:21:15.929 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-04-23T21:21:15.930 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-23T21:21:15.930 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:15.930 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:21:15.930 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:15.930 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-23T21:21:15.930 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:21:15.931 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-23T21:21:15.931 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-23T21:21:15.931 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:15.931 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-23T21:21:15.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:15.932 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-23T21:21:15.932 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:21:15.932 DEBUG:teuthology.orchestra.run.smithi111:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-04-23T21:21:15.960 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:15.960+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:15.991 INFO:tasks.ceph.osd.1:Stopped 2024-04-23T21:21:15.991 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-04-23T21:21:15.991 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:21:15.991 DEBUG:teuthology.orchestra.run.smithi111:> type ceph-objectstore-tool 2024-04-23T21:21:15.992 INFO:teuthology.orchestra.run.smithi111.stdout:mapped writes DISABLED 2024-04-23T21:21:15.992 INFO:teuthology.orchestra.run.smithi111.stdout:mapped reads DISABLED 2024-04-23T21:21:15.992 INFO:teuthology.orchestra.run.smithi111.stdout:Seed set to 1154 2024-04-23T21:21:16.033 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:16.033+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:16.045 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-23T21:21:16.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-04-23T21:21:16.072 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:16.072 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:16.072 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:16.072 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:16.083 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:16.093 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:16.093 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:16.134 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:16.135+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:16.149 INFO:teuthology.orchestra.run.smithi111.stdout:2.0 2024-04-23T21:21:16.149 INFO:teuthology.orchestra.run.smithi111.stdout:1.3 2024-04-23T21:21:16.198 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:16.198 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:16.207 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:16.251 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:16.251+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:16.310 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:16.310 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:16.319 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:16.335 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:16.336+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:16.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 2.0 --file /var/log/ceph/exp.2.0.1 2024-04-23T21:21:16.436 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:16.437+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:16.464 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:16.464 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:16.465 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:16.465 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:16.474 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:16.503 INFO:teuthology.orchestra.run.smithi111.stderr:Exporting 2.0 info 2.0( v 16'117 (0'0,16'117] local-lis/les=15/16 n=2 ec=15/15 lis/c=15/15 les/c/f=16/16/0 sis=15) 2024-04-23T21:21:16.506 INFO:teuthology.orchestra.run.smithi111.stderr:Read #2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-23T21:21:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:Read #2:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-23T21:21:16.507 INFO:teuthology.orchestra.run.smithi111.stderr:Export successful 2024-04-23T21:21:16.537 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:16.538+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:16.610 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:16.610 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:16.610 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:16.610 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:16.610 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:16.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:16.657 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:16.657+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:16.739 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:16.740+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:16.760 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:16.760 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [] 2024-04-23T21:21:16.760 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:16.763 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:21:16.772 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:16.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 2.0 2024-04-23T21:21:16.867 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:16.867+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:16.885 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:16.886 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:16.886 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:16.886 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:16.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:16.942 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:16.942+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:17.038 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:17.038 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:17.038 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:17.038 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:17.043 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:17.044+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:17.047 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:17.144 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:17.144+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:17.150 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:17.151 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:17.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:17.151 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [] 2024-04-23T21:21:17.151 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:17.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:17.245 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:17.246+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:17.274 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.274 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:17.274 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:17.274 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:17.283 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:17.367 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:17.368+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:17.426 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.426 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:17.427 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:17.427 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:17.427 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:17.435 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:17.476 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:17.477+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:17.547 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:17.547+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:17.577 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.578 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:17.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:17.578 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [] 2024-04-23T21:21:17.578 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:17.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:17.674 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:17.675+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:17.733 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.733 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:17.733 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:17.733 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:17.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:17.782 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:17.783+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:17.849 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:17.850+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:17.884 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:17.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:17.944 INFO:teuthology.orchestra.run.smithi111.stdout: marking collection for removal 2024-04-23T21:21:17.944 INFO:teuthology.orchestra.run.smithi111.stdout:setting '_remove' omap key 2024-04-23T21:21:17.944 INFO:teuthology.orchestra.run.smithi111.stdout:finish_remove_pgs 2.0_head removing 2.0 2024-04-23T21:21:17.948 INFO:teuthology.orchestra.run.smithi111.stdout:Remove successful 2024-04-23T21:21:17.950 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:17.950+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:17.995 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:17.996 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:17.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:17.996 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [] 2024-04-23T21:21:17.996 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.004 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:18.051 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:18.052+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:18.107 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.107 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:18.107 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:18.107 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:18.152 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:18.152+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:18.253 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:18.254+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:18.258 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.258 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:18.258 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:18.258 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:18.258 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:18.354 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:18.355+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:18.410 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.410 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:18.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:18.410 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [] 2024-04-23T21:21:18.410 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.418 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:18.460 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:18.460+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:18.460 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.0.1 2024-04-23T21:21:18.555 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:18.556+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:18.562 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.562 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:18.562 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:18.562 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:18.656 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:18.656+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:18.713 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:18.713 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:18.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:18.762 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:18.763+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:18.781 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:21:18.858 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:18.859+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:18.867 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:18.867 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:70 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.819363+0000", 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.0465468770000008, 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081469160000000006, 2024-04-23T21:21:18.868 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.869 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 70 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819363+0000", 2024-04-23T21:21:18.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819363+0000", 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819363+0000", 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4799999999999999e-07 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819367+0000", 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8410000000000002e-06 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819372+0000", 2024-04-23T21:21:18.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6349999999999997e-06 2024-04-23T21:21:18.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.819376+0000", 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0749999999999999e-06 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.821979+0000", 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026037529999999999 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.822031+0000", 2024-04-23T21:21:18.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2129000000000002e-05 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.822105+0000", 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3776000000000007e-05 2024-04-23T21:21:18.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.825642+0000", 2024-04-23T21:21:18.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003537203 2024-04-23T21:21:18.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.827451+0000", 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001808245 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.827484+0000", 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3031999999999998e-05 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.827509+0000", 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5778999999999998e-05 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:85 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.847890+0000", 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.0180195909999998, 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093059119999999995, 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.882 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 85 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847890+0000", 2024-04-23T21:21:18.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847890+0000", 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847891+0000", 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7599999999999997e-07 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847895+0000", 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:21:18.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847902+0000", 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9070000000000001e-06 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847907+0000", 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0610000000000003e-06 2024-04-23T21:21:18.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849100+0000", 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011928419999999999 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849156+0000", 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5723999999999997e-05 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849237+0000", 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1840000000000002e-05 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.853080+0000", 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038430309999999998 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857093+0000", 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040122680000000003 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857172+0000", 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9536999999999994e-05 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857196+0000", 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3535000000000001e-05 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:86 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.847918+0000", 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.0179917809999992, 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093273360000000003, 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 86 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847918+0000", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847918+0000", 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847918+0000", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7800000000000002e-07 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847922+0000", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7960000000000002e-06 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847934+0000", 2024-04-23T21:21:18.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1739000000000001e-05 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847939+0000", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0719999999999999e-06 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849475+0000", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015367250000000001 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849530+0000", 2024-04-23T21:21:18.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4614000000000003e-05 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849603+0000", 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2539999999999993e-05 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.853116+0000", 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035132179999999998 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857201+0000", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040855980000000002 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857224+0000", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2441000000000001e-05 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857245+0000", 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1115e-05 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.895 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:87 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.847949+0000", 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.0179607750000006, 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093527100000000002, 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.896 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 87 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847949+0000", 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847949+0000", 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.847949+0000", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1099999999999996e-07 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848000+0000", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1289000000000003e-05 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848009+0000", 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3250000000000008e-06 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848015+0000", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.353e-06 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849880+0000", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018644550000000001 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.849936+0000", 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5887999999999998e-05 2024-04-23T21:21:18.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.850010+0000", 2024-04-23T21:21:18.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4184000000000007e-05 2024-04-23T21:21:18.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.853145+0000", 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003135634 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857252+0000", 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041062989999999999 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857277+0000", 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5225e-05 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857301+0000", 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4547000000000001e-05 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:88 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.848027+0000", 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.0178822489999995, 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093176609999999997, 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 88 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848027+0000", 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848027+0000", 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848028+0000", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7299999999999996e-07 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848031+0000", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.687e-06 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848041+0000", 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3810000000000005e-06 2024-04-23T21:21:18.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.848046+0000", 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.87e-06 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.850464+0000", 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024179370000000002 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.850513+0000", 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9162999999999997e-05 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.850607+0000", 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4204999999999997e-05 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.853168+0000", 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002560611 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857305+0000", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041370210000000003 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857326+0000", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1104e-05 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.857345+0000", 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9108999999999999e-05 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.908 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.969442+0000", 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.8964671099999997, 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078239569999999994, 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 114 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969442+0000", 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969442+0000", 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969443+0000", 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2399999999999999e-07 2024-04-23T21:21:18.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969447+0000", 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8149999999999999e-06 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969453+0000", 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1140000000000002e-06 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969457+0000", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5070000000000002e-06 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.973469+0000", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040121310000000004 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.973549+0000", 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9546e-05 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.973674+0000", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012501400000000001 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.976612+0000", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029380880000000002 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977216+0000", 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060440299999999999 2024-04-23T21:21:18.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977247+0000", 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0173000000000001e-05 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977266+0000", 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9741999999999998e-05 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.915 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.969469+0000", 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.8964405610000004, 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078518149999999998, 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 115 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969469+0000", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969469+0000", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969469+0000", 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969537+0000", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7574e-05 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969545+0000", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6070000000000002e-06 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969550+0000", 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2320000000000001e-06 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.973907+0000", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00435681 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.973970+0000", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3776999999999996e-05 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.974042+0000", 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1969000000000002e-05 2024-04-23T21:21:18.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.976638+0000", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002596073 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977270+0000", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063162199999999998 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977292+0000", 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1565000000000001e-05 2024-04-23T21:21:18.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977321+0000", 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9088999999999999e-05 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.969559+0000", 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.8963500409999998, 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078104150000000002, 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 116 2024-04-23T21:21:18.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969559+0000", 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969559+0000", 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969560+0000", 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0999999999999999e-07 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969565+0000", 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1390000000000003e-06 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969571+0000", 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1e-06 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969576+0000", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6210000000000003e-06 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.974654+0000", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050777979999999997 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.974709+0000", 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5143999999999999e-05 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.974780+0000", 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1594999999999995e-05 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977169+0000", 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023891559999999999 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977326+0000", 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000156343 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977352+0000", 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6463999999999999e-05 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977370+0000", 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7544999999999999e-05 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:10.969583+0000", 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.8963264219999996, 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0078378509999999998, 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 117 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969583+0000", 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969583+0000", 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969583+0000", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.77e-07 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969587+0000", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7440000000000001e-06 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969593+0000", 2024-04-23T21:21:18.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7010000000000001e-06 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.969599+0000", 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5649999999999998e-06 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.975009+0000", 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054098439999999996 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.975054+0000", 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4526e-05 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.975125+0000", 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1298999999999996e-05 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977199+0000", 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002074354 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977373+0000", 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000173485 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977403+0000", 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9672999999999999e-05 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:10.977421+0000", 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8283000000000001e-05 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:131 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352167+0000", 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5137429080000002, 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080727669999999998, 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.934 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352167+0000", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352167+0000", 2024-04-23T21:21:18.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352167+0000", 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4000000000000002e-07 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352211+0000", 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4076000000000003e-05 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352218+0000", 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9940000000000003e-06 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352223+0000", 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0420000000000002e-06 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353051+0000", 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00082737800000000003 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353095+0000", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4733e-05 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353166+0000", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0244e-05 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356597+0000", 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034318450000000002 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360151+0000", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0035533499999999998 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360204+0000", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3103999999999998e-05 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360239+0000", 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5460999999999998e-05 2024-04-23T21:21:18.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:132 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352234+0000", 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5136757950000002, 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080978109999999999, 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.940 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 132 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352234+0000", 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352234+0000", 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352234+0000", 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352240+0000", 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9429999999999997e-06 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352246+0000", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3450000000000002e-06 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352251+0000", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6460000000000001e-06 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353514+0000", 2024-04-23T21:21:18.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012624520000000001 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353559+0000", 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5855000000000001e-05 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353631+0000", 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1496999999999997e-05 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356603+0000", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029716819999999998 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360273+0000", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003670679 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360309+0000", 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6204000000000001e-05 2024-04-23T21:21:18.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360331+0000", 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2005999999999999e-05 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:133 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352335+0000", 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5135743599999998, 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080750460000000007, 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 133 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352335+0000", 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352335+0000", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352336+0000", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8400000000000004e-07 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352340+0000", 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.578e-06 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352348+0000", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8059999999999995e-06 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352353+0000", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4049999999999999e-06 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353866+0000", 2024-04-23T21:21:18.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015120979999999999 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353911+0000", 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5846000000000001e-05 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.353985+0000", 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3943000000000007e-05 2024-04-23T21:21:18.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356608+0000", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002622245 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360365+0000", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037571739999999998 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360392+0000", 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7243e-05 2024-04-23T21:21:18.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360410+0000", 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8124000000000001e-05 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:134 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352387+0000", 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5135224220000003, 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081099329999999997, 2024-04-23T21:21:18.952 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 134 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352387+0000", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352387+0000", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352388+0000", 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7299999999999996e-07 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352429+0000", 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1797e-05 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352438+0000", 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9409999999999996e-06 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352444+0000", 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2079999999999999e-06 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354215+0000", 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017715540000000001 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354260+0000", 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4537999999999999e-05 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354329+0000", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9227999999999995e-05 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356613+0000", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022837059999999999 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360447+0000", 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003834601 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360478+0000", 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0667999999999998e-05 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360497+0000", 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9119000000000001e-05 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:135 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352456+0000", 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5134536790000004, 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081119439999999994, 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 135 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.959 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352456+0000", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352456+0000", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352456+0000", 2024-04-23T21:21:18.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0999999999999999e-07 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352462+0000", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8089999999999998e-06 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352469+0000", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352476+0000", 2024-04-23T21:21:18.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3420000000000002e-06 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354716+0000", 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002239619 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354756+0000", 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0049000000000002e-05 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.354832+0000", 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5668999999999997e-05 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356617+0000", 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017856969999999999 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360526+0000", 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039087330000000002 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360550+0000", 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4119000000000001e-05 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360568+0000", 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7659000000000002e-05 2024-04-23T21:21:18.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:136 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352485+0000", 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.513424197, 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081589899999999996, 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 136 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352485+0000", 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352485+0000", 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352486+0000", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5999999999999999e-07 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352489+0000", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5939999999999998e-06 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352500+0000", 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0891e-05 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352505+0000", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8099999999999997e-06 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355097+0000", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002592072 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355176+0000", 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8510000000000006e-05 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355311+0000", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013501 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.358533+0000", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032219990000000001 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360600+0000", 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002067794 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360627+0000", 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.652e-05 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360644+0000", 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7329999999999998e-05 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:137 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352514+0000", 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5133956140000002, 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081910920000000005, 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352514+0000", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352514+0000", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352514+0000", 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3599999999999999e-07 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352558+0000", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4218000000000003e-05 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352571+0000", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2826999999999999e-05 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352577+0000", 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5720000000000002e-06 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355612+0000", 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0030346739999999998 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355656+0000", 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4505999999999997e-05 2024-04-23T21:21:18.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.355721+0000", 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4819000000000004e-05 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.358536+0000", 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028153240000000001 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360665+0000", 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021290079999999999 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360688+0000", 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2849000000000001e-05 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.360705+0000", 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6858999999999999e-05 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:138 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352586+0000", 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5133230989999999, 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008670522, 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.977 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 138 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352586+0000", 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352586+0000", 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352587+0000", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9999999999999998e-07 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352590+0000", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5549999999999998e-06 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352598+0000", 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0899999999999999e-06 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352602+0000", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6539999999999998e-06 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356090+0000", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034876059999999999 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356164+0000", 2024-04-23T21:21:18.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4238000000000003e-05 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356287+0000", 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012286900000000001 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.359009+0000", 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027224910000000001 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361211+0000", 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022011539999999999 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361237+0000", 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6271999999999999e-05 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361257+0000", 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0092999999999999e-05 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:139 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352610+0000", 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5132990800000004, 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090274510000000006, 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 139 2024-04-23T21:21:18.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352610+0000", 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352610+0000", 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352611+0000", 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352614+0000", 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4790000000000001e-06 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352620+0000", 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352625+0000", 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6480000000000002e-06 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356657+0000", 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004031728 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356704+0000", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6894999999999997e-05 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.356775+0000", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1044999999999995e-05 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361290+0000", 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045156959999999996 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361590+0000", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030001099999999998 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361618+0000", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8201999999999999e-05 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361638+0000", 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9357000000000001e-05 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.988 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352633+0000", 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.51327677, 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090450530000000008, 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.989 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 140 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352633+0000", 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352633+0000", 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352633+0000", 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.27e-07 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352681+0000", 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7877000000000003e-05 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352690+0000", 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7509999999999997e-06 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352695+0000", 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2830000000000001e-06 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.357011+0000", 2024-04-23T21:21:18.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043155509999999999 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.357065+0000", 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4493999999999997e-05 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.357135+0000", 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9595e-05 2024-04-23T21:21:18.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361321+0000", 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0041867079999999999 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361641+0000", 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031972700000000001 2024-04-23T21:21:18.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361662+0000", 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0531999999999999e-05 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361678+0000", 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6107999999999999e-05 2024-04-23T21:21:18.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4213.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:11.352705+0000", 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5132048539999996, 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090124139999999998, 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4213", 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.192:0/4267745442", 2024-04-23T21:21:18.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 141 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352705+0000", 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352705+0000", 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352705+0000", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9999999999999998e-07 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352709+0000", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7929999999999999e-06 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352715+0000", 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7690000000000002e-06 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.352719+0000", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.358565+0000", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058456960000000001 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.358609+0000", 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3634000000000003e-05 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:18.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.358677+0000", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8781000000000004e-05 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361343+0000", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026660020000000002 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361682+0000", 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00033818100000000001 2024-04-23T21:21:19.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361702+0000", 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0874999999999999e-05 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:11.361717+0000", 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4635e-05 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:19.001 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:19.003 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:19.004+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:19.004 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:19.103 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:19.104+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:19.108 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:19.108 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:19.117 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:19.204 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:19.205+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:19.260 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:19.261 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:19.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:19.305 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:19.306+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:19.374 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:19.374 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:19.383 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:19.406 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:19.407+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:19.485 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:19.486 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:19.486 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:19.486 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:19.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:19.508 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:19.509+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:19.596 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:19.606 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:19.608 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:19.609+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:19.613 INFO:teuthology.orchestra.run.smithi111.stdout:get_pg_num_history pg_num_history pg_num_history(e17 pg_nums {1={14=8},2={15=1}} deleted_pools ) 2024-04-23T21:21:19.613 INFO:teuthology.orchestra.run.smithi111.stdout:Importing pgid 2.0 2024-04-23T21:21:19.617 INFO:teuthology.orchestra.run.smithi111.stdout:Write #2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-04-23T21:21:19.617 INFO:teuthology.orchestra.run.smithi111.stdout:snapset 0=[]:{} 2024-04-23T21:21:19.618 INFO:teuthology.orchestra.run.smithi111.stdout:Write #2:30306672:devicehealth::main.db.0000000000000000:head# 2024-04-23T21:21:19.618 INFO:teuthology.orchestra.run.smithi111.stdout:snapset 0=[]:{} 2024-04-23T21:21:19.619 INFO:teuthology.orchestra.run.smithi111.stdout:write_pg epoch 16 info 2.0( v 16'117 (0'0,16'117] local-lis/les=15/16 n=2 ec=15/15 lis/c=15/15 les/c/f=16/16/0 sis=15) 2024-04-23T21:21:19.620 INFO:teuthology.orchestra.run.smithi111.stdout:Import successful 2024-04-23T21:21:19.709 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:19.709 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:19.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:19.709 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [] 2024-04-23T21:21:19.709 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:19.710 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:19.710+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:19.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:19.782 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:21:19.839 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:19.839+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:19.865 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:19.865 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:19.865 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:19.865 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:19.874 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:19.916 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:19.916+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:20.012 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:20.013+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:20.017 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:20.018 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:20.018 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:20.018 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:20.018 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:20.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:20.113 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:20.113+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:20.128 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:20.129 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:20.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:20.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [] 2024-04-23T21:21:20.129 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:20.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /var/log/ceph/exp.2.0.1 2024-04-23T21:21:20.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:20.214 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:20.214+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:20.251 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:20.252 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:20.346 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:20.346+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:20.404 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.404 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:20.404 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:20.404 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:20.404 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:20.454 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:20.454+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:20.517 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:20.518+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:4 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:17.783354+0000", 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.7723778100000001, 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00015118999999999999, 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.555 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783354+0000", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783354+0000", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783357+0000", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8660000000000002e-06 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783361+0000", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0849999999999999e-06 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783369+0000", 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6190000000000002e-06 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783377+0000", 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0330000000000008e-06 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783421+0000", 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3720999999999997e-05 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783506+0000", 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4865999999999997e-05 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:5 3.1a 3:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19)", 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:17.783644+0000", 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.7720887049999998, 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014225199999999999, 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:20.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 5 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783644+0000", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783644+0000", 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783644+0000", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3200000000000005e-07 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783647+0000", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5380000000000001e-06 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783653+0000", 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-06 2024-04-23T21:21:20.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783657+0000", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8680000000000001e-06 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783701+0000", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3449999999999999e-05 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:17.783786+0000", 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5273999999999997e-05 2024-04-23T21:21:20.562 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7 3.1a 3:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:18.774506+0000", 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.781226355, 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001063149, 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.563 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774506+0000", 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774506+0000", 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774508+0000", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.818e-06 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774517+0000", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.003e-06 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774533+0000", 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5998000000000002e-05 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774545+0000", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2738999999999999e-05 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774595+0000", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9969000000000002e-05 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.774680+0000", 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4120000000000001e-05 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.775439+0000", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00075952599999999995 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.775489+0000", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9897e-05 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:18.775569+0000", 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0079000000000004e-05 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.790608+0000", 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.76512419600000003, 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037534119999999998, 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 9 2024-04-23T21:21:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790608+0000", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790608+0000", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790610+0000", 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6199999999999999e-06 2024-04-23T21:21:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790616+0000", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1909999999999997e-06 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790630+0000", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4102e-05 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790639+0000", 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0769999999999997e-06 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790689+0000", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0439999999999998e-05 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.790773+0000", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3419999999999998e-05 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.794269+0000", 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034956779999999999 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.794309+0000", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0114000000000003e-05 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.794361+0000", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2769999999999998e-05 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:10 3.1c e21/18 3:3831950c:::rbd_directory:head v 21'1, mlcod=0'0)", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.797360+0000", 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.75837186000000001, 2024-04-23T21:21:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073132799999999995, 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797360+0000", 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797360+0000", 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797361+0000", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5699999999999999e-07 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797367+0000", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3339999999999998e-06 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797387+0000", 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0132e-05 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797395+0000", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2970000000000004e-06 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797453+0000", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8023000000000003e-05 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797484+0000", 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0320000000000001e-05 2024-04-23T21:21:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798044+0000", 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056062500000000003 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798092+0000", 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999997e-05 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:13 3.1c e21/18 3:397f8f07:::rbd_object_map.1118a1ef21e9:head v 21'2, mlcod=0'0)", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.809226+0000", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.74650588299999998, 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022726579999999999, 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809226+0000", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809226+0000", 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809227+0000", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8999999999999995e-07 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809234+0000", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5400000000000001e-06 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809247+0000", 2024-04-23T21:21:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2969000000000001e-05 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809252+0000", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7790000000000001e-06 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809306+0000", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3570999999999999e-05 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809329+0000", 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2932e-05 2024-04-23T21:21:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811452+0000", 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021234259999999999 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811499+0000", 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650999999999998e-05 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:23 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.821288+0000", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.73444472699999996, 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047530899999999999, 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 23 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821288+0000", 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821288+0000", 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821291+0000", 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.993e-06 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821306+0000", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5823000000000001e-05 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821343+0000", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6263000000000002e-05 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821358+0000", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5075999999999999e-05 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821486+0000", 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012861099999999999 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821624+0000", 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013723199999999999 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.821763+0000", 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013931100000000001 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.585 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:20.618 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:20.618+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:20.688 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:20.688 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:20.740 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:20.741+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:20.800 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.800 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:20.800 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:20.800 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:20.800 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.809 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:20.851 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:20.851+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:20.920 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:20.921+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:20.953 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:20.953 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.159321625, 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033009100000000002, 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.954 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795206+0000", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4489999999999999e-06 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795212+0000", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7780000000000004e-06 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795221+0000", 2024-04-23T21:21:20.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795228+0000", 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.362e-06 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795293+0000", 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749e-05 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:20.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795297+0000", 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3800000000000004e-06 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.796968+0000", 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001670988 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797050+0000", 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2093000000000002e-05 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797200+0000", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014947699999999999 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797917+0000", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00071695299999999999 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798432+0000", 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051571899999999999 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798477+0000", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5114999999999997e-05 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798503+0000", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5797000000000001e-05 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:10 3.1c e21/18 ondisk, result = 0)", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:20.960 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.1561764489999999, 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016184399999999999, 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798349+0000", 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.066e-06 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798355+0000", 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9089999999999997e-06 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798365+0000", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0812000000000001e-05 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798373+0000", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5909999999999998e-06 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798421+0000", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8517999999999999e-05 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798429+0000", 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4699999999999996e-06 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798509+0000", 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0477999999999998e-05 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:11 3.1f e21/18 3:fa654f18:::rbd_id.image_client.0:head v 21'1, mlcod=0'0)", 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:20.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.15360201, 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092272100000000004, 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:20.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800934+0000", 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1405e-05 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800948+0000", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3986e-05 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800956+0000", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8509999999999999e-06 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801027+0000", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1483000000000007e-05 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801054+0000", 2024-04-23T21:21:20.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6638000000000001e-05 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801799+0000", 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000745041 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801845+0000", 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5964999999999998e-05 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:12 3.17 e21/18 3:ec4c83eb:::rbd_header.1118a1ef21e9:head v 21'1, mlcod=0'0)", 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.1476637489999999, 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064791399999999998, 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806861+0000", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6899999999999996e-07 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806866+0000", 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1660000000000002e-06 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806878+0000", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1347e-05 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806905+0000", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6999999999999999e-05 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806969+0000", 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4498000000000005e-05 2024-04-23T21:21:20.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806981+0000", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1982e-05 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.807479+0000", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049812700000000001 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.807508+0000", 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9025000000000001e-05 2024-04-23T21:21:20.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:13 3.1c 3:397f8f07:::rbd_object_map.1118a1ef21e9:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.1457417809999999, 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069253029999999998, 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:20.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 13 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808783+0000", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9000000000000005e-07 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808789+0000", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4700000000000001e-06 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808801+0000", 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2146e-05 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808809+0000", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2409999999999995e-06 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808894+0000", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5331e-05 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809008+0000", 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113244 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809145+0000", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013712800000000001 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811616+0000", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024716170000000002 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815545+0000", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003928633 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815637+0000", 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1798000000000004e-05 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815708+0000", 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0704999999999995e-05 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:13 3.1c e21/18 ondisk, result = 0)", 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.1429882739999999, 2024-04-23T21:21:20.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5480000000000002e-05, 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5199999999999996e-07 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811540+0000", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.642e-06 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811550+0000", 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0077e-05 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811558+0000", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0120000000000005e-06 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811606+0000", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8134999999999999e-05 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811613+0000", 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4899999999999997e-06 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811621+0000", 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4719999999999995e-06 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:25 3.17 e21/18 3:ec4c83eb:::rbd_header.1118a1ef21e9:head v 21'2, mlcod=0'0)", 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.1319143600000001, 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063668700000000004, 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:20.982 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4499999999999996e-07 2024-04-23T21:21:20.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822616+0000", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7230000000000001e-06 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822628+0000", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1432e-05 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822651+0000", 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3954000000000001e-05 2024-04-23T21:21:20.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822704+0000", 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2099999999999999e-05 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822727+0000", 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3839999999999999e-05 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:20.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.823212+0000", 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000485036 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.823246+0000", 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3856999999999998e-05 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:20.986 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:20.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:21.021 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:21.021+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:21.067 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:21.098 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:21.098 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:21.122 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:21.122+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:21.129 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:21.129 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:21.130 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:21.130 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:21.130 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:21.130 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:21.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:21.159 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:21.159 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:21.189 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:21.190 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:21.217 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:21.217 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:21.223 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:21.224+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:21.234 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:21.234 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:21.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:21.242 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:21.242 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:21.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:21.336 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:21.337+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:21.394 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:21.394 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:21.402 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:21.443 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:21.444+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:21.525 INFO:teuthology.orchestra.run.smithi111.stdout:skipping zero size read 2024-04-23T21:21:21.526 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:21.526+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:21.540 INFO:teuthology.orchestra.run.smithi111.stdout:4 write 0x880ae0f thru 0x880fd71 (0x4f63 bytes) 2024-04-23T21:21:21.549 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:21.549 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:21.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:21.560 INFO:teuthology.orchestra.run.smithi111.stdout:6 writesame 0xea3e50f thru 0xea3fd3d data_size 0x1(0x182f bytes) 2024-04-23T21:21:21.560 INFO:teuthology.orchestra.run.smithi111.stdout:7 punch from 0x33d7238 to 0x33e2bdf, (0xb9a7 bytes) 2024-04-23T21:21:21.560 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0x35626dd 2024-04-23T21:21:21.573 INFO:teuthology.orchestra.run.smithi111.stdout:8 trunc from 0xea3fd3e to 0x35626dd 2024-04-23T21:21:21.584 INFO:teuthology.orchestra.run.smithi111.stdout:9 punch from 0x2fff398 to 0x3001967, (0x25cf bytes) 2024-04-23T21:21:21.603 INFO:teuthology.orchestra.run.smithi111.stdout:10 write 0xdab709d thru 0xdac4eca (0xde2e bytes) 2024-04-23T21:21:21.607 INFO:teuthology.orchestra.run.smithi111.stdout:11 read 0xac147fd thru 0xac17c58 (0x345c bytes) 2024-04-23T21:21:21.607 INFO:teuthology.orchestra.run.smithi111.stdout:12 write 0x6528bf1 thru 0x652a035 (0x1445 bytes) 2024-04-23T21:21:21.622 INFO:teuthology.orchestra.run.smithi111.stdout:13 trunc from 0xdac4ecb to 0x2d583a1 2024-04-23T21:21:21.641 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:21.642+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:21.652 INFO:teuthology.orchestra.run.smithi111.stdout:14 writesame 0xbe682a2 thru 0xbe76006 data_size 0x13(0xdd65 bytes) 2024-04-23T21:21:21.652 INFO:teuthology.orchestra.run.smithi111.stdout:15 writesame 0x3fae8a5 thru 0x3faeae4 data_size 0x12(0x240 bytes) 2024-04-23T21:21:21.655 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:21.655 INFO:teuthology.orchestra.run.smithi111.stdout:17 write 0xb8d7156 thru 0xb8e3379 (0xc224 bytes) 2024-04-23T21:21:21.659 INFO:teuthology.orchestra.run.smithi111.stdout:18 write 0x3427364 thru 0x34300a8 (0x8d45 bytes) 2024-04-23T21:21:21.674 INFO:teuthology.orchestra.run.smithi111.stdout:19 trunc from 0xbe76007 to 0x132a499 2024-04-23T21:21:21.686 INFO:teuthology.orchestra.run.smithi111.stdout:20 read 0x9b3e5d thru 0x9bb3c9 (0x756d bytes) 2024-04-23T21:21:21.696 INFO:teuthology.orchestra.run.smithi111.stdout:22 write 0x3fe57bf thru 0x3ff398a (0xe1cc bytes) 2024-04-23T21:21:21.699 INFO:teuthology.orchestra.run.smithi111.stdout:23 punch from 0x315eaf2 to 0x316969a, (0xaba8 bytes) 2024-04-23T21:21:21.706 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:21.707 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:21.712 INFO:teuthology.orchestra.run.smithi111.stdout:24 write 0xaf23ff5 thru 0xaf28dbe (0x4dca bytes) 2024-04-23T21:21:21.715 INFO:teuthology.orchestra.run.smithi111.stdout:25 read 0xa7a3001 thru 0xa7ab6b6 (0x86b6 bytes) 2024-04-23T21:21:21.715 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:21.715 INFO:teuthology.orchestra.run.smithi111.stdout:28 write 0x531a80 thru 0x5345b8 (0x2b39 bytes) 2024-04-23T21:21:21.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:21.718 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:21.718 INFO:teuthology.orchestra.run.smithi111.stdout:30 punch from 0x56eb5fd to 0x56f86dd, (0xd0e0 bytes) 2024-04-23T21:21:21.718 INFO:teuthology.orchestra.run.smithi111.stdout:32 write 0x5cb51dd thru 0x5cb6c99 (0x1abd bytes) 2024-04-23T21:21:21.721 INFO:teuthology.orchestra.run.smithi111.stdout:33 write 0x859f7f2 thru 0x85a616a (0x6979 bytes) 2024-04-23T21:21:21.725 INFO:teuthology.orchestra.run.smithi111.stdout:37 read 0x605760c thru 0x605e9a5 (0x739a bytes) 2024-04-23T21:21:21.733 INFO:teuthology.orchestra.run.smithi111.stdout:38 write 0xc3d0e20 thru 0xc3d7414 (0x65f5 bytes) 2024-04-23T21:21:21.735 INFO:teuthology.orchestra.run.smithi111.stdout:39 read 0x6f57893 thru 0x6f5f3c0 (0x7b2e bytes) 2024-04-23T21:21:21.735 INFO:teuthology.orchestra.run.smithi111.stdout:40 writesame 0xae6a728 thru 0xae6f803 data_size 0x73(0x50dc bytes) 2024-04-23T21:21:21.736 INFO:teuthology.orchestra.run.smithi111.stdout:41 read 0x4c09e83 thru 0x4c173dc (0xd55a bytes) 2024-04-23T21:21:21.743 INFO:teuthology.orchestra.run.smithi111.stdout:42 write 0xdec491d thru 0xdec7cc1 (0x33a5 bytes) 2024-04-23T21:21:21.753 INFO:teuthology.orchestra.run.smithi111.stdout:43 write 0xed5a402 thru 0xed693fc (0xeffb bytes) 2024-04-23T21:21:21.756 INFO:teuthology.orchestra.run.smithi111.stdout:44 read 0xa2fa718 thru 0xa2fb13e (0xa27 bytes) 2024-04-23T21:21:21.756 INFO:teuthology.orchestra.run.smithi111.stdout:46 write 0x91f5c0e thru 0x92014c7 (0xb8ba bytes) 2024-04-23T21:21:21.757 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:21.758+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:21.760 INFO:teuthology.orchestra.run.smithi111.stdout:47 write 0x56c79c thru 0x56ee06 (0x266b bytes) 2024-04-23T21:21:21.761 INFO:teuthology.orchestra.run.smithi111.stdout:48 write 0x311f7b thru 0x318de5 (0x6e6b bytes) 2024-04-23T21:21:21.765 INFO:teuthology.orchestra.run.smithi111.stdout:49 read 0x42bee40 thru 0x42c118a (0x234b bytes) 2024-04-23T21:21:21.765 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:21.765 INFO:teuthology.orchestra.run.smithi111.stdout:52 writesame 0x4658f80 thru 0x4667b6b data_size 0x7(0xebec bytes) 2024-04-23T21:21:21.771 INFO:teuthology.orchestra.run.smithi111.stdout:53 write 0x71c2261 thru 0x71cfc85 (0xda25 bytes) 2024-04-23T21:21:21.774 INFO:teuthology.orchestra.run.smithi111.stdout:54 writesame 0x6003e3b thru 0x600d3ec data_size 0x9(0x95b2 bytes) 2024-04-23T21:21:21.778 INFO:teuthology.orchestra.run.smithi111.stdout:55 read 0xdd727ef thru 0xdd74b5c (0x236e bytes) 2024-04-23T21:21:21.778 INFO:teuthology.orchestra.run.smithi111.stdout:56 write 0xe880d14 thru 0xe88df64 (0xd251 bytes) 2024-04-23T21:21:21.783 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:21.783 INFO:teuthology.orchestra.run.smithi111.stdout:58 read 0x4b83dcd thru 0x4b86f28 (0x315c bytes) 2024-04-23T21:21:21.783 INFO:teuthology.orchestra.run.smithi111.stdout:59 write 0x78d5660 thru 0x78e1c27 (0xc5c8 bytes) 2024-04-23T21:21:21.787 INFO:teuthology.orchestra.run.smithi111.stdout:60 punch from 0x571bee3 to 0x571ca92, (0xbaf bytes) 2024-04-23T21:21:21.787 INFO:teuthology.orchestra.run.smithi111.stdout:61 read 0x3a86829 thru 0x3a89e7e (0x3656 bytes) 2024-04-23T21:21:21.787 INFO:teuthology.orchestra.run.smithi111.stdout:62 clone 1 order 24 su 65536 sc 7 2024-04-23T21:21:21.798 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:21:21.842 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:21.842+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:21.860 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:21.860 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:21.869 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:21.929 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:21.929+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:21.999 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.014 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.015 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:22.059 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:22.059+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:22.118 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.118 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:22.131 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:22.131+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:22.232 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:22.232+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:22.270 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.270 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:22.333 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:22.334+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:22.422 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.422 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:22.433 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:22.434+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:22.533 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.534 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.534 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:22.535+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:22.542 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:22.634 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:22.635+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:22.644 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.644 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:22.735 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:22.736+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:22.757 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.757 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:22.786 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:21:22.835 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:22.836+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:22.877 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:22.877 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:22.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:22.936 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:22.937+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:22.986 INFO:teuthology.orchestra.run.smithi111.stdout:63 write 0x3f1bcdc thru 0x3f24f57 (0x927c bytes) 2024-04-23T21:21:23.000 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:23.001 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:23.003 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.004 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.004 INFO:teuthology.orchestra.run.smithi111.stdout:66 read 0x903d20e thru 0x903ecd0 (0x1ac3 bytes) 2024-04-23T21:21:23.004 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xa7a4d97 2024-04-23T21:21:23.009 INFO:teuthology.orchestra.run.smithi111.stdout:67 trunc from 0xed693fd to 0xa7a4d97 2024-04-23T21:21:23.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:23.043 INFO:teuthology.orchestra.run.smithi111.stdout:69 write 0xa21a0b2 thru 0xa2264ef (0xc43e bytes) 2024-04-23T21:21:23.045 INFO:teuthology.orchestra.run.smithi111.stdout:70 read 0x3e9e19a thru 0x3ea2cb4 (0x4b1b bytes) 2024-04-23T21:21:23.045 INFO:teuthology.orchestra.run.smithi111.stdout:73 read 0x577e60e thru 0x578de2e (0xf821 bytes) 2024-04-23T21:21:23.050 INFO:teuthology.orchestra.run.smithi111.stdout:74 write 0xa97cd44 thru 0xa98708b (0xa348 bytes) 2024-04-23T21:21:23.051 INFO:teuthology.orchestra.run.smithi111.stdout:75 punch from 0x896b92a to 0x8976c73, (0xb349 bytes) 2024-04-23T21:21:23.052 INFO:teuthology.orchestra.run.smithi111.stdout:76 writesame 0x5e37e29 thru 0x5e44587 data_size 0x35(0xc75f bytes) 2024-04-23T21:21:23.056 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:23.057+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:23.066 INFO:teuthology.orchestra.run.smithi111.stdout:77 writesame 0xc7d72c6 thru 0xc7e1c86 data_size 0x1(0xa9c1 bytes) 2024-04-23T21:21:23.073 INFO:teuthology.orchestra.run.smithi111.stdout:78 write 0xd000217 thru 0xd004180 (0x3f6a bytes) 2024-04-23T21:21:23.082 INFO:teuthology.orchestra.run.smithi111.stdout:80 trunc from 0xd004181 to 0x60e4429 2024-04-23T21:21:23.111 INFO:teuthology.orchestra.run.smithi111.stdout:82 punch from 0x3905f35 to 0x3906500, (0x5cb bytes) 2024-04-23T21:21:23.111 INFO:teuthology.orchestra.run.smithi111.stdout:83 punch from 0xcc4fa6 to 0xcd1971, (0xc9cb bytes) 2024-04-23T21:21:23.123 INFO:teuthology.orchestra.run.smithi111.stdout:84 write 0xbf3b97a thru 0xbf48ddd (0xd464 bytes) 2024-04-23T21:21:23.126 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.126 INFO:teuthology.orchestra.run.smithi111.stdout:86 write 0x737b478 thru 0x73813b3 (0x5f3c bytes) 2024-04-23T21:21:23.130 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.130 INFO:teuthology.orchestra.run.smithi111.stdout:88 writesame 0x1056752 thru 0x1062816 data_size 0x1(0xc0c5 bytes) 2024-04-23T21:21:23.131 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:23.131 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:23.138 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:23.139+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:23.140 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:23.145 INFO:teuthology.orchestra.run.smithi111.stdout:89 read 0x5631295 thru 0x5631c97 (0xa03 bytes) 2024-04-23T21:21:23.152 INFO:teuthology.orchestra.run.smithi111.stdout:90 write 0xd18a1a7 thru 0xd199ec7 (0xfd21 bytes) 2024-04-23T21:21:23.154 INFO:teuthology.orchestra.run.smithi111.stdout:92 read 0xb6273f4 thru 0xb630910 (0x951d bytes) 2024-04-23T21:21:23.154 INFO:teuthology.orchestra.run.smithi111.stdout:93 write 0xc89506b thru 0xc896317 (0x12ad bytes) 2024-04-23T21:21:23.156 INFO:teuthology.orchestra.run.smithi111.stdout:94 writesame 0x17f7100 thru 0x17fb02c data_size 0x9(0x3f2d bytes) 2024-04-23T21:21:23.158 INFO:teuthology.orchestra.run.smithi111.stdout:95 read 0xa553fd8 thru 0xa55f2fe (0xb327 bytes) 2024-04-23T21:21:23.158 INFO:teuthology.orchestra.run.smithi111.stdout:100 punch from 0xaee6e48 to 0xaeebcad, (0x4e65 bytes) 2024-04-23T21:21:23.163 INFO:teuthology.orchestra.run.smithi111.stdout:101 write 0xd659195 thru 0xd65d9b7 (0x4823 bytes) 2024-04-23T21:21:23.164 INFO:teuthology.orchestra.run.smithi111.stdout:102 write 0x5a0746 thru 0x5a7953 (0x720e bytes) 2024-04-23T21:21:23.165 INFO:teuthology.orchestra.run.smithi111.stdout:103 write 0x9bc67d6 thru 0x9bc9c36 (0x3461 bytes) 2024-04-23T21:21:23.167 INFO:teuthology.orchestra.run.smithi111.stdout:105 write 0x584060f thru 0x584715e (0x6b50 bytes) 2024-04-23T21:21:23.168 INFO:teuthology.orchestra.run.smithi111.stdout:107 write 0x15a175 thru 0x166650 (0xc4dc bytes) 2024-04-23T21:21:23.170 INFO:teuthology.orchestra.run.smithi111.stdout:108 write 0x7cb559e thru 0x7cbc696 (0x70f9 bytes) 2024-04-23T21:21:23.183 INFO:teuthology.orchestra.run.smithi111.stdout:112 trunc from 0xd65d9b8 to 0x3aeac6d 2024-04-23T21:21:23.200 INFO:teuthology.orchestra.run.smithi111.stdout:113 punch from 0x4adc13 to 0x4b45e7, (0x69d4 bytes) 2024-04-23T21:21:23.200 INFO:teuthology.orchestra.run.smithi111.stdout:114 read 0xba0e41 thru 0xba51a3 (0x4363 bytes) 2024-04-23T21:21:23.200 INFO:teuthology.orchestra.run.smithi111.stdout:115 read 0x167f58c thru 0x168e5b7 (0xf02c bytes) 2024-04-23T21:21:23.203 INFO:teuthology.orchestra.run.smithi111.stdout:116 trunc from 0x3aeac6d to 0x22736dd 2024-04-23T21:21:23.211 INFO:teuthology.orchestra.run.smithi111.stdout:117 read 0x21f870b thru 0x22020fa (0x99f0 bytes) 2024-04-23T21:21:23.222 INFO:teuthology.orchestra.run.smithi111.stdout:118 trunc from 0x22736dd to 0xa44d018 2024-04-23T21:21:23.227 INFO:teuthology.orchestra.run.smithi111.stdout:119 writesame 0x6399b74 thru 0x63a1dd5 data_size 0x3(0x8262 bytes) 2024-04-23T21:21:23.232 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.238 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:23.240 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:23.239+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:23.242 INFO:teuthology.orchestra.run.smithi111.stdout:122 write 0xd5881ee thru 0xd58f57c (0x738f bytes) 2024-04-23T21:21:23.245 INFO:teuthology.orchestra.run.smithi111.stdout:123 writesame 0xb58797d thru 0xb58af90 data_size 0x2(0x3614 bytes) 2024-04-23T21:21:23.250 INFO:teuthology.orchestra.run.smithi111.stdout:125 read 0xba33a29 thru 0xba40e19 (0xd3f1 bytes) 2024-04-23T21:21:23.250 INFO:teuthology.orchestra.run.smithi111.stdout:128 write 0xae09318 thru 0xae0c9aa (0x3693 bytes) 2024-04-23T21:21:23.253 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.253 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.253 INFO:teuthology.orchestra.run.smithi111.stdout:133 write 0xc4a089a thru 0xc4a2272 (0x19d9 bytes) 2024-04-23T21:21:23.257 INFO:teuthology.orchestra.run.smithi111.stdout:134 writesame 0x9444b3f thru 0x9448b0d data_size 0x37(0x3fcf bytes) 2024-04-23T21:21:23.275 INFO:teuthology.orchestra.run.smithi111.stdout:135 trunc from 0xd58f57d to 0x5eecc4 2024-04-23T21:21:23.291 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:23.291 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:23.292 INFO:teuthology.orchestra.run.smithi111.stdout:136 read 0x4c2985 thru 0x4c6e35 (0x44b1 bytes) 2024-04-23T21:21:23.292 INFO:teuthology.orchestra.run.smithi111.stdout:137 read 0x8654c thru 0x88794 (0x2249 bytes) 2024-04-23T21:21:23.292 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.293 INFO:teuthology.orchestra.run.smithi111.stdout:139 punch from 0x1085ce to 0x10de3e, (0x5870 bytes) 2024-04-23T21:21:23.293 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.300 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:23.305 INFO:teuthology.orchestra.run.smithi111.stdout:142 write 0x6921f5a thru 0x692febb (0xdf62 bytes) 2024-04-23T21:21:23.315 INFO:teuthology.orchestra.run.smithi111.stdout:143 writesame 0xa17f60b thru 0xa18cd86 data_size 0x6(0xd77c bytes) 2024-04-23T21:21:23.323 INFO:teuthology.orchestra.run.smithi111.stdout:144 write 0xbaa1afc thru 0xbaa6ebf (0x53c4 bytes) 2024-04-23T21:21:23.325 INFO:teuthology.orchestra.run.smithi111.stdout:145 writesame 0x5477eca thru 0x54814b7 data_size 0x3(0x95ee bytes) 2024-04-23T21:21:23.340 INFO:teuthology.orchestra.run.smithi111.stdout:146 trunc from 0xbaa6ec0 to 0x2cab2ff 2024-04-23T21:21:23.340 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:23.340+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:23.353 INFO:teuthology.orchestra.run.smithi111.stdout:147 writesame 0x28cd7d9 thru 0x28d6f47 data_size 0x1(0x976f bytes) 2024-04-23T21:21:23.372 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:23.372 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:23.372 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:23.383 INFO:teuthology.orchestra.run.smithi111.stdout:148 writesame 0xd91d14f thru 0xd922243 data_size 0x5(0x50f5 bytes) 2024-04-23T21:21:23.390 INFO:teuthology.orchestra.run.smithi111.stdout:150 write 0xee5d19c thru 0xee64e51 (0x7cb6 bytes) 2024-04-23T21:21:23.391 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:23.391 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:23.404 INFO:teuthology.orchestra.run.smithi111.stdout:151 trunc from 0xee64e52 to 0x6378b7f 2024-04-23T21:21:23.409 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:23.409 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:23.410 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:23.410 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:23.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:23.424 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:23.424 INFO:teuthology.orchestra.run.smithi111.stdout:153 writesame 0x3b738f0 thru 0x3b770ae data_size 0x43(0x37bf bytes) 2024-04-23T21:21:23.426 INFO:teuthology.orchestra.run.smithi111.stdout:154 write 0x1a48099 thru 0x1a4ed51 (0x6cb9 bytes) 2024-04-23T21:21:23.428 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:23.428 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:23.438 INFO:teuthology.orchestra.run.smithi111.stdout:155 writesame 0xb27802c thru 0xb283cfc data_size 0x1(0xbcd1 bytes) 2024-04-23T21:21:23.447 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:23.448 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:23.449 INFO:teuthology.orchestra.run.smithi111.stdout:156 trunc from 0xb283cfd to 0x183e87e 2024-04-23T21:21:23.462 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:23.463+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:23.466 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:23.466 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:23.481 INFO:teuthology.orchestra.run.smithi111.stdout:157 write 0xaae9f44 thru 0xaaf6a2f (0xcaec bytes) 2024-04-23T21:21:23.486 INFO:teuthology.orchestra.run.smithi111.stdout:158 read 0x6c05447 thru 0x6c0bd18 (0x68d2 bytes) 2024-04-23T21:21:23.486 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:23.486 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:23.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:23.495 INFO:teuthology.orchestra.run.smithi111.stdout:159 write 0xe178a31 thru 0xe178b73 (0x143 bytes) 2024-04-23T21:21:23.503 INFO:teuthology.orchestra.run.smithi111.stdout:160 write 0xe7e9097 thru 0xe7ecb18 (0x3a82 bytes) 2024-04-23T21:21:23.506 INFO:teuthology.orchestra.run.smithi111.stdout:161 clone 2 order 22 su 32768 sc 7 2024-04-23T21:21:23.530 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:23.530 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:23.530 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:23.530 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:23.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:23.579 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:23.580+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:23.644 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:23.645+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:23.681 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:23.682 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:23.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:23.745 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:23.745+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_add_image in=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0438824039999997, 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033009100000000002, 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795202+0000", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795206+0000", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4489999999999999e-06 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795212+0000", 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7780000000000004e-06 2024-04-23T21:21:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795221+0000", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795228+0000", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.362e-06 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795293+0000", 2024-04-23T21:21:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749e-05 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.795297+0000", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3800000000000004e-06 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.796968+0000", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001670988 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797050+0000", 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2093000000000002e-05 2024-04-23T21:21:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797200+0000", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00014947699999999999 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.797917+0000", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00071695299999999999 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798432+0000", 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00051571899999999999 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798477+0000", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5114999999999997e-05 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798503+0000", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5797000000000001e-05 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:10 3.1c e21/18 ondisk, result = 0)", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0407372280000002, 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016184399999999999, 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798348+0000", 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798349+0000", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.066e-06 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798355+0000", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9089999999999997e-06 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798365+0000", 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0812000000000001e-05 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798373+0000", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5909999999999998e-06 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798421+0000", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8517999999999999e-05 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798429+0000", 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4699999999999996e-06 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.798509+0000", 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0477999999999998e-05 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:11 3.1f e21/18 3:fa654f18:::rbd_id.image_client.0:head v 21'1, mlcod=0'0)", 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0381627890000003, 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092272100000000004, 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800922+0000", 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800934+0000", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1405e-05 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800948+0000", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3986e-05 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.800956+0000", 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8509999999999999e-06 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801027+0000", 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1483000000000007e-05 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801054+0000", 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6638000000000001e-05 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801799+0000", 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000745041 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.801845+0000", 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5964999999999998e-05 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:12 3.17 e21/18 3:ec4c83eb:::rbd_header.1118a1ef21e9:head v 21'1, mlcod=0'0)", 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0322245280000004, 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064791399999999998, 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806860+0000", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806861+0000", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6899999999999996e-07 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806866+0000", 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1660000000000002e-06 2024-04-23T21:21:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806878+0000", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1347e-05 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806905+0000", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6999999999999999e-05 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806969+0000", 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4498000000000005e-05 2024-04-23T21:21:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.806981+0000", 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1982e-05 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.807479+0000", 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049812700000000001 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.807508+0000", 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9025000000000001e-05 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:13 3.1c 3:397f8f07:::rbd_object_map.1118a1ef21e9:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.03030256, 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069253029999999998, 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 13 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808782+0000", 2024-04-23T21:21:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808783+0000", 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9000000000000005e-07 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808789+0000", 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4700000000000001e-06 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808801+0000", 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2146e-05 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808809+0000", 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2409999999999995e-06 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.808894+0000", 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5331e-05 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809008+0000", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113244 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.809145+0000", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013712800000000001 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811616+0000", 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024716170000000002 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815545+0000", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003928633 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815637+0000", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1798000000000004e-05 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.815708+0000", 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0704999999999995e-05 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:13 3.1c e21/18 ondisk, result = 0)", 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0275490530000004, 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5480000000000002e-05, 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811536+0000", 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5199999999999996e-07 2024-04-23T21:21:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811540+0000", 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.642e-06 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811550+0000", 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0077e-05 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811558+0000", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0120000000000005e-06 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811606+0000", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8134999999999999e-05 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811613+0000", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4899999999999997e-06 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.811621+0000", 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4719999999999995e-06 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:25 3.17 e21/18 3:ec4c83eb:::rbd_header.1118a1ef21e9:head v 21'2, mlcod=0'0)", 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0164751389999997, 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063668700000000004, 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822610+0000", 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4499999999999996e-07 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822616+0000", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7230000000000001e-06 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822628+0000", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1432e-05 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822651+0000", 2024-04-23T21:21:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3954000000000001e-05 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822704+0000", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2099999999999999e-05 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.822727+0000", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3839999999999999e-05 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.823212+0000", 2024-04-23T21:21:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000485036 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:19.823246+0000", 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3856999999999998e-05 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1512 3.5 3:a2abbaec:::rbd_data.1118c9f76dd8.0000000000000001:head [stat,truncate 3670016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.451826+0000", 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.38725902499999998, 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038571009999999999, 2024-04-23T21:21:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1512 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451826+0000", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451826+0000", 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451827+0000", 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0410000000000001e-06 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451834+0000", 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1500000000000002e-06 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451850+0000", 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6407000000000001e-05 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451860+0000", 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.696e-06 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451942+0000", 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2377000000000001e-05 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.452041+0000", 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8958999999999999e-05 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3", 2024-04-23T21:21:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.452191+0000", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000150094 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453283+0000", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001091472 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455610+0000", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023275050000000001 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455653+0000", 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2904000000000001e-05 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455683+0000", 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9496e-05 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1516 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:head [stat,truncate 3604480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.451887+0000", 2024-04-23T21:21:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.38719780300000001, 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037768620000000002, 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1516 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451887+0000", 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451887+0000", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451888+0000", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451890+0000", 2024-04-23T21:21:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451898+0000", 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0120000000000005e-06 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451904+0000", 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6389999999999997e-06 2024-04-23T21:21:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.451977+0000", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2540999999999995e-05 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.452058+0000", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1056000000000005e-05 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.452191+0000", 2024-04-23T21:21:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013386800000000001 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453308+0000", 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011166990000000001 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455574+0000", 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022663010000000001 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455620+0000", 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5945999999999997e-05 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455664+0000", 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3371999999999997e-05 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:1512 3.5 e22/18 ondisk, result = 0)", 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.453195+0000", 2024-04-23T21:21:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.385890179, 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7369999999999998e-05, 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453195+0000", 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453195+0000", 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453195+0000", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7499999999999999e-07 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453200+0000", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4059999999999998e-06 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453209+0000", 2024-04-23T21:21:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6169999999999996e-06 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453217+0000", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4229999999999999e-06 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453269+0000", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1631000000000001e-05 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453277+0000", 2024-04-23T21:21:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4889999999999995e-06 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.453292+0000", 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4929000000000001e-05 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:1516 3.1d e22/18 ondisk, result = 0)", 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.455495+0000", 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.38358937300000001, 2024-04-23T21:21:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017380100000000001, 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455495+0000", 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455495+0000", 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455496+0000", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0999999999999998e-07 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455500+0000", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455511+0000", 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0427e-05 2024-04-23T21:21:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455517+0000", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3709999999999996e-06 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455564+0000", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7348000000000001e-05 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455570+0000", 2024-04-23T21:21:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1529999999999996e-06 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.455669+0000", 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9667000000000006e-05 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1519 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'70, mlcod=22'68)", 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.464275+0000", 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.374809958, 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064509699999999997, 2024-04-23T21:21:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464275+0000", 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464275+0000", 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464276+0000", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.17e-06 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464283+0000", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.306e-06 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464295+0000", 2024-04-23T21:21:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1263999999999999e-05 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464301+0000", 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9689999999999999e-06 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464348+0000", 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7145999999999999e-05 2024-04-23T21:21:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464364+0000", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6140000000000001e-05 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464889+0000", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052509200000000001 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.464920+0000", 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1010000000000003e-05 2024-04-23T21:21:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1527 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'71, mlcod=22'69)", 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.479191+0000", 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.35989357300000002, 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00077361599999999997, 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479191+0000", 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479191+0000", 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479192+0000", 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9099999999999991e-07 2024-04-23T21:21:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479199+0000", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6120000000000004e-06 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479212+0000", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3037000000000001e-05 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479219+0000", 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2019999999999998e-06 2024-04-23T21:21:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479276+0000", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6586999999999999e-05 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479300+0000", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4705000000000001e-05 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479931+0000", 2024-04-23T21:21:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063026800000000002 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.479965+0000", 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4214000000000001e-05 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1536 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'72, mlcod=22'70)", 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.483986+0000", 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.35509826, 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080276100000000003, 2024-04-23T21:21:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.483986+0000", 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.483986+0000", 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.483987+0000", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6300000000000005e-07 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.483992+0000", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484002+0000", 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7690000000000008e-06 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484007+0000", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7639999999999996e-06 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484057+0000", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9808999999999997e-05 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484078+0000", 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1914999999999999e-05 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484755+0000", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00067633700000000005 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484789+0000", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4483999999999997e-05 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1537 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'73, mlcod=22'70)", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.484812+0000", 2024-04-23T21:21:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.35427315199999998, 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073729399999999999, 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484812+0000", 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484812+0000", 2024-04-23T21:21:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484812+0000", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1300000000000001e-07 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484816+0000", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0380000000000001e-06 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484824+0000", 2024-04-23T21:21:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9030000000000004e-06 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484829+0000", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484867+0000", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8303999999999998e-05 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.484886+0000", 2024-04-23T21:21:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9119000000000001e-05 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.485504+0000", 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00061736500000000004 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.485549+0000", 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5392999999999997e-05 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1541 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'74, mlcod=22'72)", 2024-04-23T21:21:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.492966+0000", 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.346119071, 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080944999999999997, 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492966+0000", 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492966+0000", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492967+0000", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.105e-06 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492973+0000", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492983+0000", 2024-04-23T21:21:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397000000000001e-05 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.492993+0000", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9830000000000008e-06 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.493054+0000", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0477999999999999e-05 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.493075+0000", 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1691e-05 2024-04-23T21:21:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.493725+0000", 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064959800000000004 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.493775+0000", 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0120999999999997e-05 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1550 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'75, mlcod=22'73)", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.497714+0000", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.34137098900000001, 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080674000000000002, 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497714+0000", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497714+0000", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497715+0000", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0299999999999998e-07 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497720+0000", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7389999999999996e-06 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497731+0000", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0658e-05 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497736+0000", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.186e-06 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497789+0000", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3087999999999997e-05 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.497810+0000", 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069000000000001e-05 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.498471+0000", 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066022300000000004 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.498521+0000", 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9974e-05 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1552 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'76, mlcod=22'74)", 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.500746+0000", 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.33833918299999999, 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00059453399999999997, 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500746+0000", 2024-04-23T21:21:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500746+0000", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500746+0000", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6899999999999997e-07 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500751+0000", 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6269999999999999e-06 2024-04-23T21:21:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500760+0000", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6490000000000003e-06 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500765+0000", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8790000000000002e-06 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500803+0000", 2024-04-23T21:21:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7950000000000001e-05 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.500818+0000", 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5004000000000001e-05 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.501293+0000", 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00047461200000000002 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.501340+0000", 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7243999999999997e-05 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:1561 3.1f e22/18 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head v 22'77, mlcod=22'75)", 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.505953+0000", 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.33313173800000001, 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00089481900000000004, 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505953+0000", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505953+0000", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505954+0000", 2024-04-23T21:21:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505960+0000", 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.737e-06 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505970+0000", 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5889999999999992e-06 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.505975+0000", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9110000000000001e-06 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.506027+0000", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2376999999999997e-05 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.506050+0000", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3135000000000001e-05 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.506801+0000", 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00075093100000000004 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.506848+0000", 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6533999999999999e-05 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:23.926 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:23.927 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone1 intact 2024-04-23T21:21:23.928 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:23.928 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:23.928+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:24.012 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-04-23T21:21:24.028 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:24.029+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:24.073 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:24.073 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:24.129 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:24.129+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:24.224 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:24.225 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.230 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:24.231+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:24.233 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:24.265 INFO:teuthology.orchestra.run.smithi111.stdout:163 write 0x3735411 thru 0x3743dc7 (0xe9b7 bytes) 2024-04-23T21:21:24.318 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.318 INFO:teuthology.orchestra.run.smithi111.stdout:165 read 0xa54c51 thru 0xa62c34 (0xdfe4 bytes) 2024-04-23T21:21:24.319 INFO:teuthology.orchestra.run.smithi111.stdout:167 write 0x9480023 thru 0x949001f (0xfffd bytes) 2024-04-23T21:21:24.331 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:24.331+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:168 punch from 0x419c97c to 0x41a9c0b, (0xd28f bytes) 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:171 read 0xabe191e thru 0xabe424f (0x2932 bytes) 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:173 read 0xd447eba thru 0xd44b928 (0x3a6f bytes) 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:174 read 0xd1007f6 thru 0xd1057bc (0x4fc7 bytes) 2024-04-23T21:21:24.332 INFO:teuthology.orchestra.run.smithi111.stdout:175 write 0xdd14a08 thru 0xdd16727 (0x1d20 bytes) 2024-04-23T21:21:24.341 INFO:teuthology.orchestra.run.smithi111.stdout:177 read 0x4f70fbe thru 0x4f78874 (0x78b7 bytes) 2024-04-23T21:21:24.341 INFO:teuthology.orchestra.run.smithi111.stdout:181 writesame 0x8add8a thru 0x8b2e94 data_size 0x1(0x510b bytes) 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1633 3.a 3:535385f0:::rbd_children:head [call rbd.add_child in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.54229141800000003, 2024-04-23T21:21:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023809069999999998, 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1633 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850633+0000", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4160000000000001e-06 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850640+0000", 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6919999999999996e-06 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850654+0000", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4722000000000001e-05 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850661+0000", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8059999999999997e-06 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850710+0000", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9081e-05 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850714+0000", 2024-04-23T21:21:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8650000000000003e-06 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852078+0000", 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013635400000000001 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852122+0000", 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4078e-05 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852245+0000", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012301 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852783+0000", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053843000000000001 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852956+0000", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017254799999999999 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852989+0000", 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2802000000000002e-05 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.853013+0000", 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3917000000000002e-05 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:1633 3.a e23/18 ondisk, result = 0)", 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.852885+0000", 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.54003761299999997, 2024-04-23T21:21:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132402, 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852885+0000", 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852885+0000", 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852886+0000", 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7299999999999995e-07 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852889+0000", 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3289999999999998e-06 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852897+0000", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.593e-06 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852903+0000", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5879999999999997e-06 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852947+0000", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4589999999999998e-05 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852953+0000", 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5300000000000004e-06 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.853018+0000", 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5098999999999995e-05 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1719 3.a 3:53a44285:::rbd_data.1118a1ef21e9.000000000000000e:head [sparse-read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.066809+0000", 2024-04-23T21:21:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.32611428799999997, 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026593500000000002, 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1719 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066809+0000", 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066809+0000", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066811+0000", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9539999999999998e-06 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066819+0000", 2024-04-23T21:21:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6710000000000005e-06 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066839+0000", 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999999e-05 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066848+0000", 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7000000000000003e-06 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.066971+0000", 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000122848 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067075+0000", 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103562 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1754 3.10 3:0cf8ba48:::rbd_data.1118a1ef21e9.0000000000000031:head [sparse-read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.067875+0000", 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.32504842, 2024-04-23T21:21:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018966299999999999, 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1754 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067875+0000", 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067875+0000", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067876+0000", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.16e-07 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067880+0000", 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5909999999999998e-06 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067895+0000", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.467e-05 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067903+0000", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9929999999999995e-06 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.067969+0000", 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6045999999999998e-05 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.068064+0000", 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5446999999999996e-05 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.412 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.413 INFO:teuthology.orchestra.run.smithi111.stdout:182 trunc from 0xe7ecb19 to 0x2224b4 2024-04-23T21:21:24.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:24.459 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:24.460+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:24.481 INFO:teuthology.orchestra.run.smithi111.stdout:183 write 0xea0425d thru 0xea0a7e4 (0x6588 bytes) 2024-04-23T21:21:24.489 INFO:teuthology.orchestra.run.smithi111.stdout:184 writesame 0x68270e5 thru 0x6834cfb data_size 0x7(0xdc17 bytes) 2024-04-23T21:21:24.499 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.500 INFO:teuthology.orchestra.run.smithi111.stdout:186 write 0x93e761e thru 0x93ee7fe (0x71e1 bytes) 2024-04-23T21:21:24.506 INFO:teuthology.orchestra.run.smithi111.stdout:187 punch from 0x383841b to 0x383e83d, (0x6422 bytes) 2024-04-23T21:21:24.506 INFO:teuthology.orchestra.run.smithi111.stdout:188 writesame 0x111bc47 thru 0x111e6a2 data_size 0x2(0x2a5c bytes) 2024-04-23T21:21:24.509 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.509 INFO:teuthology.orchestra.run.smithi111.stdout:190 punch from 0xafdc28b to 0xafddcc3, (0x1a38 bytes) 2024-04-23T21:21:24.509 INFO:teuthology.orchestra.run.smithi111.stdout:191 writesame 0x702fc45 thru 0x703bf0e data_size 0x3(0xc2ca bytes) 2024-04-23T21:21:24.520 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.528 INFO:teuthology.orchestra.run.smithi111.stdout:193 trunc from 0xea0a7e5 to 0x78601c1 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3271 3.15 3:ab9bcb8b:::rbd_data.11183c92f5c.000000000000003e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.532651+0000", 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00086563299999999998, 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00088343299999999998, 2024-04-23T21:21:24.532 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3271 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532651+0000", 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532651+0000", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532653+0000", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.251e-06 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532658+0000", 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3650000000000003e-06 2024-04-23T21:21:24.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532671+0000", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2989e-05 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532679+0000", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6769999999999995e-06 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532725+0000", 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6683e-05 2024-04-23T21:21:24.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532777+0000", 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1309e-05 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3", 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532838+0000", 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1389000000000005e-05 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.536 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:3270 3.7 e23/18 3:e3c9f743:::rbd_data.11183c92f5c.000000000000003d:head v 23'40, mlcod=23'38)", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.532911+0000", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.000605891, 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066650599999999998, 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532911+0000", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532911+0000", 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532912+0000", 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532916+0000", 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7099999999999998e-06 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532924+0000", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.391e-06 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532935+0000", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1342000000000001e-05 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.532987+0000", 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2222000000000002e-05 2024-04-23T21:21:24.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.533009+0000", 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1797000000000002e-05 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-04-23T21:21:24.540 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.541 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:24.532+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:24.542 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:24.550 INFO:teuthology.orchestra.run.smithi111.stdout:195 write 0x35dd8d2 thru 0x35de3b7 (0xae6 bytes) 2024-04-23T21:21:24.554 INFO:teuthology.orchestra.run.smithi111.stdout:197 writesame 0x4b0fca4 thru 0x4b14822 data_size 0xb(0x4b7f bytes) 2024-04-23T21:21:24.561 INFO:teuthology.orchestra.run.smithi111.stdout:198 write 0x50c7d5b thru 0x50d12fe (0x95a4 bytes) 2024-04-23T21:21:24.568 INFO:teuthology.orchestra.run.smithi111.stdout:200 read 0x77534a3 thru 0x775dc40 (0xa79e bytes) 2024-04-23T21:21:24.580 INFO:teuthology.orchestra.run.smithi111.stdout:201 write 0xb0779f7 thru 0xb079fc2 (0x25cc bytes) 2024-04-23T21:21:24.586 INFO:teuthology.orchestra.run.smithi111.stdout:202 read 0x7b440fa thru 0x7b4c083 (0x7f8a bytes) 2024-04-23T21:21:24.586 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.587 INFO:teuthology.orchestra.run.smithi111.stdout:204 write 0x364f28e thru 0x365ceb7 (0xdc2a bytes) 2024-04-23T21:21:24.593 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xc62696a 2024-04-23T21:21:24.595 INFO:teuthology.orchestra.run.smithi111.stdout:205 trunc from 0xb079fc3 to 0xc62696a 2024-04-23T21:21:24.610 INFO:teuthology.orchestra.run.smithi111.stdout:207 write 0xd6b58d8 thru 0xd6c488f (0xefb8 bytes) 2024-04-23T21:21:24.615 INFO:teuthology.orchestra.run.smithi111.stdout:208 writesame 0xad8c664 thru 0xad8e5f5 data_size 0x9(0x1f92 bytes) 2024-04-23T21:21:24.626 INFO:teuthology.orchestra.run.smithi111.stdout:209 trunc from 0xd6c4890 to 0x875c8d0 2024-04-23T21:21:24.633 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:24.633+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:24.642 INFO:teuthology.orchestra.run.smithi111.stdout:210 punch from 0x82d0524 to 0x82d6530, (0x600c bytes) 2024-04-23T21:21:24.643 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.647 INFO:teuthology.orchestra.run.smithi111.stdout:215 trunc from 0x875c8d0 to 0x4b7be2b 2024-04-23T21:21:24.661 INFO:teuthology.orchestra.run.smithi111.stdout:216 read 0x4b62352 thru 0x4b68746 (0x63f5 bytes) 2024-04-23T21:21:24.661 INFO:teuthology.orchestra.run.smithi111.stdout:217 write 0x7400e2 thru 0x74dcd2 (0xdbf1 bytes) 2024-04-23T21:21:24.662 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.671 INFO:teuthology.orchestra.run.smithi111.stdout:219 trunc from 0x4b7be2b to 0xbb900c3 2024-04-23T21:21:24.676 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.677 INFO:teuthology.orchestra.run.smithi111.stdout:221 read 0x35b6b0 thru 0x360890 (0x51e1 bytes) 2024-04-23T21:21:24.677 INFO:teuthology.orchestra.run.smithi111.stdout:225 write 0x7d2892c thru 0x7d2c0ba (0x378f bytes) 2024-04-23T21:21:24.679 INFO:teuthology.orchestra.run.smithi111.stdout:227 read 0x5f230c3 thru 0x5f29112 (0x6050 bytes) 2024-04-23T21:21:24.679 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.688 INFO:teuthology.orchestra.run.smithi111.stdout:229 trunc from 0xbb900c3 to 0x3fc6725 2024-04-23T21:21:24.691 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.691 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:24.691 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:24.691 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:24.691 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:24.716 INFO:teuthology.orchestra.run.smithi111.stdout:230 write 0xcd9cfc3 thru 0xcda1cc0 (0x4cfe bytes) 2024-04-23T21:21:24.723 INFO:teuthology.orchestra.run.smithi111.stdout:232 trunc from 0xcda1cc1 to 0x978df42 2024-04-23T21:21:24.734 INFO:teuthology.orchestra.run.smithi111.stdout:233 read 0x5f41b75 thru 0x5f4bd22 (0xa1ae bytes) 2024-04-23T21:21:24.744 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:24.745+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:24.745 INFO:teuthology.orchestra.run.smithi111.stdout:234 trunc from 0x978df42 to 0x13aa62a 2024-04-23T21:21:24.757 INFO:teuthology.orchestra.run.smithi111.stdout:236 read 0x3a26d9 thru 0x3a41a4 (0x1acc bytes) 2024-04-23T21:21:24.757 INFO:teuthology.orchestra.run.smithi111.stdout:237 read 0xfe7a4 thru 0xfef66 (0x7c3 bytes) 2024-04-23T21:21:24.757 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:24.757 INFO:teuthology.orchestra.run.smithi111.stdout:241 write 0xcf37d1 thru 0xd0127f (0xdaaf bytes) 2024-04-23T21:21:24.759 INFO:teuthology.orchestra.run.smithi111.stdout:242 read 0x58d824 thru 0x59c00d (0xe7ea bytes) 2024-04-23T21:21:24.759 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xd5c4fcd 2024-04-23T21:21:24.774 INFO:teuthology.orchestra.run.smithi111.stdout:243 trunc from 0x13aa62a to 0xd5c4fcd 2024-04-23T21:21:24.779 INFO:teuthology.orchestra.run.smithi111.stdout:244 write 0xabd2525 thru 0xabd7768 (0x5244 bytes) 2024-04-23T21:21:24.793 INFO:teuthology.orchestra.run.smithi111.stdout:245 trunc from 0xd5c4fcd to 0x449aa24 2024-04-23T21:21:24.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:21:24.806 INFO:teuthology.orchestra.run.smithi111.stdout:246 write 0x801338 thru 0x80e296 (0xcf5f bytes) 2024-04-23T21:21:24.806 INFO:teuthology.orchestra.run.smithi111.stdout:247 trunc from 0x449aa24 to 0x3cd67da 2024-04-23T21:21:24.812 INFO:teuthology.orchestra.run.smithi111.stdout:248 read 0x1e6d08 thru 0x1f080b (0x9b04 bytes) 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3197 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2916352~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39370039299999998, 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024247707, 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3197 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420534+0000", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.074e-06 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420544+0000", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3370000000000006e-06 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420549+0000", 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9030000000000003e-06 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444634+0000", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024085517000000001 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444713+0000", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444780+0000", 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6797000000000004e-05 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3198 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2981888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39367678499999997, 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024356404000000002, 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3198 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420556+0000", 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420558+0000", 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420564+0000", 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7089999999999999e-06 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420568+0000", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6280000000000004e-06 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444802+0000", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024234169999999999 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444844+0000", 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444912+0000", 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8367999999999993e-05 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39365783199999999, 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:21:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39363046899999998, 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:21:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39361622000000002, 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:21:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:21:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39359397699999998, 2024-04-23T21:21:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39352849099999998, 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:21:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:21:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39350274899999999, 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:21:24.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39347873999999999, 2024-04-23T21:21:24.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:21:24.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:21:24.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39345618100000002, 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.857 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39343637799999998, 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.858 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:21:24.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39341689200000002, 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:21:24.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:24.866 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39339709299999998, 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:21:24.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39337739100000002, 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:21:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:24.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:21:24.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39332126499999998, 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:24.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:21:24.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.393284044, 2024-04-23T21:21:24.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:21:24.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39323373099999998, 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:21:24.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:21:24.887 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.393200836, 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.39317853200000003, 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:24.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.393158116, 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:24.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:24.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:24.900 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:24.901 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:24.903 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:24.903+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:24.903 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:24.903 INFO:teuthology.orchestra.run.smithi111.stdout:249 write 0x3d47285 thru 0x3d4dad3 (0x684f bytes) 2024-04-23T21:21:24.903 INFO:teuthology.orchestra.run.smithi111.stdout:251 write 0x225cb33 thru 0x2266451 (0x991f bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:252 punch from 0x254862b to 0x2548ae2, (0x4b7 bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:253 punch from 0x175b76e to 0x1766168, (0xa9fa bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:254 write 0x2af6f55 thru 0x2b00cf1 (0x9d9d bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:255 write 0x52bfaa3 thru 0x52c3788 (0x3ce6 bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:256 punch from 0x194bc8a to 0x1957f37, (0xc2ad bytes) 2024-04-23T21:21:24.904 INFO:teuthology.orchestra.run.smithi111.stdout:257 clone 3 order 21 su 524288 sc 6 2024-04-23T21:21:25.002 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:25.003+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:25.007 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:25.007 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:25.007 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:25.007 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:25.016 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:25.103 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:25.103+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:25.120 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:25.120 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:25.120 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:25.129 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:25.202 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:25.202 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:21:25.202 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T21:21:25.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:21:25.214 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:25.213+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1467 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.387293+0000", 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.8879991029999998, 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001747373, 2024-04-23T21:21:25.276 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1467 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387293+0000", 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387293+0000", 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387295+0000", 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9460000000000001e-06 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387305+0000", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0264e-05 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387327+0000", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1103000000000001e-05 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387336+0000", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4530000000000008e-06 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387395+0000", 2024-04-23T21:21:25.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8943999999999997e-05 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387477+0000", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2053999999999997e-05 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.387650+0000", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000172604 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.388508+0000", 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00085880600000000004 2024-04-23T21:21:25.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.388974+0000", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00046565199999999999 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.389006+0000", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2330999999999999e-05 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.389041+0000", 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4215999999999999e-05 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.281 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1476 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.392333+0000", 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.882959818, 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0016651179999999999, 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1476 2024-04-23T21:21:25.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392333+0000", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392333+0000", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392334+0000", 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.085e-06 2024-04-23T21:21:25.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392339+0000", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8919999999999997e-06 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392353+0000", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.345e-05 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392360+0000", 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.221e-06 2024-04-23T21:21:25.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392405+0000", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4447000000000003e-05 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392467+0000", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2261000000000006e-05 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392687+0000", 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00022008499999999999 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393722+0000", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0010348 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393939+0000", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021710800000000001 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393968+0000", 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8968000000000001e-05 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393998+0000", 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9801000000000001e-05 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1477 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.392378+0000", 2024-04-23T21:21:25.287 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.8829143720000001, 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0019719920000000001, 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1477 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392378+0000", 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392378+0000", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392378+0000", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392381+0000", 2024-04-23T21:21:25.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7190000000000001e-06 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392388+0000", 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.392393+0000", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6849999999999998e-06 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393007+0000", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00061397999999999997 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393092+0000", 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4958999999999998e-05 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.393416+0000", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00032361899999999998 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.394167+0000", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00075139100000000002 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.394250+0000", 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2397000000000004e-05 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.394307+0000", 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7259999999999997e-05 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.394350+0000", 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2944000000000001e-05 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1527 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.293 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.478472+0000", 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.7968205080000001, 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001706625, 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1527 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478472+0000", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478472+0000", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478474+0000", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3779999999999999e-06 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478485+0000", 2024-04-23T21:21:25.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0761000000000001e-05 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478506+0000", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0720000000000002e-05 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478515+0000", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7659999999999993e-06 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478592+0000", 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6703000000000001e-05 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.478712+0000", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000119843 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.479030+0000", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000318466 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.479835+0000", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00080494300000000002 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.480062+0000", 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00022652999999999999 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.480122+0000", 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9812000000000003e-05 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.480178+0000", 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6703000000000002e-05 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1536 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.483278+0000", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.792014175, 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00194746, 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1536 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483278+0000", 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483278+0000", 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483279+0000", 2024-04-23T21:21:25.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:21:25.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483283+0000", 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3089999999999997e-06 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483294+0000", 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0529e-05 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483299+0000", 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3870000000000003e-06 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483341+0000", 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2021999999999998e-05 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483429+0000", 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7829000000000002e-05 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483818+0000", 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00038911700000000002 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485097+0000", 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0012784960000000001 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485132+0000", 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5410000000000001e-05 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485185+0000", 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2729999999999998e-05 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485226+0000", 2024-04-23T21:21:25.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0751000000000003e-05 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1537 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.483321+0000", 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.7919716960000001, 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0023966780000000002, 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.306 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1537 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483321+0000", 2024-04-23T21:21:25.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483321+0000", 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483321+0000", 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0900000000000002e-07 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483323+0000", 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3209999999999999e-06 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483328+0000", 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8489999999999997e-06 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.483332+0000", 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999998e-06 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.484202+0000", 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00087025700000000004 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.484287+0000", 2024-04-23T21:21:25.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4796999999999995e-05 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.484701+0000", 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000414473 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485469+0000", 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00076835100000000002 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485628+0000", 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015875700000000001 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485679+0000", 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0398000000000001e-05 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.485717+0000", 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8776000000000003e-05 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1552 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.500208+0000", 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.775083942, 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001675597, 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.313 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1552 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500208+0000", 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500208+0000", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500209+0000", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.33e-07 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500215+0000", 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5269999999999998e-06 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500226+0000", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1374e-05 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500232+0000", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7649999999999999e-06 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500279+0000", 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7463999999999998e-05 2024-04-23T21:21:25.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500370+0000", 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0477999999999997e-05 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.500623+0000", 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00025284599999999998 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.501417+0000", 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079467800000000005 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.501824+0000", 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00040606099999999999 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.501857+0000", 2024-04-23T21:21:25.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3627000000000002e-05 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.501884+0000", 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6843999999999999e-05 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1561 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.505269+0000", 2024-04-23T21:21:25.319 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.770023803, 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018168889999999999, 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1561 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505269+0000", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505269+0000", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505269+0000", 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:25.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505273+0000", 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.104e-06 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505283+0000", 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9720000000000004e-06 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505288+0000", 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9880000000000004e-06 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505333+0000", 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4783999999999997e-05 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505415+0000", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1409000000000006e-05 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.505780+0000", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00036527899999999998 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.506964+0000", 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0011841379999999999 2024-04-23T21:21:25.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.507024+0000", 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0467000000000002e-05 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.507058+0000", 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3503000000000001e-05 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.507085+0000", 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7446000000000001e-05 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1567 3.1f 3:f8eb1b8b:::rbd_object_map.1118c9f76dd8:head [read 0~40] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.799565+0000", 2024-04-23T21:21:25.326 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.475727096, 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0017840359999999999, 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1567 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799565+0000", 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799565+0000", 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799567+0000", 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.598e-06 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799574+0000", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.435e-06 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799590+0000", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5848000000000001e-05 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799599+0000", 2024-04-23T21:21:25.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8580000000000006e-06 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799663+0000", 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3552000000000004e-05 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op must wait for map", 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.799670+0000", 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1080000000000004e-06 2024-04-23T21:21:25.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.801163+0000", 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0014933100000000001 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.801246+0000", 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3207999999999993e-05 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.801349+0000", 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000103119 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2231 3.1f 3:fc2de94d:::rbd_data.11183c92f5c.0000000000000023:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write 1277952~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.323234+0000", 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.95205879999999998, 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0025780809999999999, 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.332 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2231 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323234+0000", 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323234+0000", 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323235+0000", 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3880000000000001e-06 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323260+0000", 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5435e-05 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323273+0000", 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2765999999999999e-05 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323280+0000", 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9e-06 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323337+0000", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6997000000000003e-05 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323487+0000", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014962999999999999 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.323685+0000", 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000198352 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.324379+0000", 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00069393600000000003 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.325749+0000", 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001369884 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.325785+0000", 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6363999999999999e-05 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.325812+0000", 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6429000000000001e-05 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3258 3.1f 3:fc2de94d:::rbd_data.11183c92f5c.0000000000000023:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1209886~2530] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.504322+0000", 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.77096996100000004, 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002639836, 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3258 2024-04-23T21:21:25.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504322+0000", 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504322+0000", 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504324+0000", 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6449999999999999e-06 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504330+0000", 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.536e-06 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504341+0000", 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1817000000000001e-05 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504347+0000", 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0519999999999998e-06 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504391+0000", 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3744e-05 2024-04-23T21:21:25.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504449+0000", 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7343000000000003e-05 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.504532+0000", 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3798e-05 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.505135+0000", 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00060282700000000001 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.506870+0000", 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001735066 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.506923+0000", 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2389999999999998e-05 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.506962+0000", 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9617999999999999e-05 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3268 3.1f 3:fc2de94d:::rbd_data.11183c92f5c.0000000000000023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.532600+0000", 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.74269264599999996, 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0025189850000000001, 2024-04-23T21:21:25.345 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3268 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532600+0000", 2024-04-23T21:21:25.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532600+0000", 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532601+0000", 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7e-06 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532607+0000", 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.755e-06 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532624+0000", 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6901999999999998e-05 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532631+0000", 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2110000000000001e-06 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532690+0000", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8576000000000003e-05 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532742+0000", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1833999999999999e-05 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 5", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532801+0000", 2024-04-23T21:21:25.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9556999999999999e-05 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.533357+0000", 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00055541399999999995 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.535060+0000", 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0017028880000000001 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.535091+0000", 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1733000000000003e-05 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.535119+0000", 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7415000000000001e-05 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3269 3.18 e23/18 3:1ace6d32:::rbd_data.11183c92f5c.0000000000000024:head v 23'7, mlcod=23'5)", 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.74234461600000001, 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0030814610000000002, 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532949+0000", 2024-04-23T21:21:25.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1999999999999998e-07 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532955+0000", 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7209999999999997e-06 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532966+0000", 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0848e-05 2024-04-23T21:21:25.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532975+0000", 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0040000000000005e-06 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.533041+0000", 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5458999999999998e-05 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.533060+0000", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9726000000000001e-05 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.535997+0000", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0029370870000000001 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.536029+0000", 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1795999999999998e-05 2024-04-23T21:21:25.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3286 3.12 3:4e946cf6:::rbd_data.11183c92f5c.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3889362~2790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.72208392899999996, 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028384349999999998, 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3286 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553209+0000", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6700000000000003e-07 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553214+0000", 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7430000000000002e-06 2024-04-23T21:21:25.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553224+0000", 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0155e-05 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553229+0000", 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3149999999999999e-06 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553272+0000", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2330999999999998e-05 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553353+0000", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0822000000000006e-05 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553440+0000", 2024-04-23T21:21:25.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7670000000000006e-05 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.554029+0000", 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00058887499999999999 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.555990+0000", 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0019609739999999999 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.556021+0000", 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0491999999999999e-05 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.556047+0000", 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6291e-05 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3308 3.1b 3:decf9eed:::rbd_data.11183c92f5c.000000000000002f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1245184~8131] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.585748+0000", 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.68954441200000005, 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0025190529999999998, 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.364 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3308 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585748+0000", 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585748+0000", 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585750+0000", 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5850000000000001e-06 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585760+0000", 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0801e-05 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585775+0000", 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4756999999999999e-05 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585782+0000", 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0110000000000003e-06 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585831+0000", 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9122999999999997e-05 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.585926+0000", 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4685999999999994e-05 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.586016+0000", 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0452999999999996e-05 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.586644+0000", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00062717800000000005 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.588200+0000", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0015561360000000001 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.588235+0000", 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5800999999999998e-05 2024-04-23T21:21:25.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.588267+0000", 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1522000000000001e-05 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3335 3.3 e23/18 3:c6d3752e:::rbd_data.11183c92f5c.0000000000000034:head v 23'6, mlcod=23'4)", 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.614534+0000", 2024-04-23T21:21:25.370 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.660758556, 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001745117, 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614534+0000", 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614534+0000", 2024-04-23T21:21:25.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614535+0000", 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5499999999999996e-07 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614560+0000", 2024-04-23T21:21:25.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5409e-05 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614573+0000", 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3162999999999999e-05 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614580+0000", 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7510000000000002e-06 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614627+0000", 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.744e-05 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.614642+0000", 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.43e-05 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.616236+0000", 2024-04-23T21:21:25.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0015943649999999999 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.616279+0000", 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2734000000000001e-05 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3343 3.1b 3:decf9eed:::rbd_data.11183c92f5c.000000000000002f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.629386+0000", 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.64590643999999997, 2024-04-23T21:21:25.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0020205869999999999, 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3343 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629386+0000", 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629386+0000", 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629387+0000", 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.592e-06 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629394+0000", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0209999999999996e-06 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629408+0000", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4961e-05 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629418+0000", 2024-04-23T21:21:25.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3209999999999994e-06 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629482+0000", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3961999999999995e-05 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629539+0000", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7525000000000002e-05 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.629602+0000", 2024-04-23T21:21:25.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2228000000000001e-05 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630021+0000", 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041931899999999997 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.631312+0000", 2024-04-23T21:21:25.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0012908640000000001 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.631364+0000", 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2484000000000001e-05 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.631406+0000", 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.231e-05 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.381 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3344 3.3 e23/18 3:c6d3752e:::rbd_data.11183c92f5c.0000000000000034:head v 23'7, mlcod=23'5)", 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.630642+0000", 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.64464988899999998, 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0020039620000000002, 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630642+0000", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630642+0000", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630643+0000", 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2399999999999996e-07 2024-04-23T21:21:25.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630649+0000", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2249999999999999e-06 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630657+0000", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3720000000000005e-06 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630664+0000", 2024-04-23T21:21:25.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6669999999999999e-06 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630707+0000", 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3133999999999997e-05 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.630728+0000", 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1695000000000002e-05 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.632618+0000", 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018894809999999999 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.632646+0000", 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8464000000000001e-05 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.386 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3365 3.3 e23/18 3:c75763a2:::rbd_data.11183c92f5c.0000000000000021:head v 23'9, mlcod=23'7)", 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.62385433300000004, 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0029539560000000002, 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651439+0000", 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9100000000000002e-07 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651446+0000", 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2989999999999999e-06 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651456+0000", 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5089999999999999e-06 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651460+0000", 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.729e-06 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651505+0000", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4481000000000003e-05 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651518+0000", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2707999999999999e-05 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.654357+0000", 2024-04-23T21:21:25.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028396369999999999 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.654392+0000", 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4702000000000001e-05 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3439 3.12 3:4e946cf6:::rbd_data.11183c92f5c.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.748269+0000", 2024-04-23T21:21:25.391 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.527023566, 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0020161300000000001, 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3439 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748269+0000", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748269+0000", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748270+0000", 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.316e-06 2024-04-23T21:21:25.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748280+0000", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4490000000000005e-06 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748294+0000", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4467000000000001e-05 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748301+0000", 2024-04-23T21:21:25.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.481e-06 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748360+0000", 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8349e-05 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748418+0000", 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7837999999999997e-05 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:25.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748496+0000", 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8778999999999999e-05 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.748967+0000", 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00047042999999999998 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.750219+0000", 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001252558 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.750252+0000", 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2190000000000002e-05 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.750285+0000", 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3272999999999998e-05 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.397 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:25.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:25.399 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:25.399+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:25.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:25.504 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:25.504 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 23, 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:21:23.792364+0000", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:21:16.757901+0000", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:20:59.765723+0000", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5804032, 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "nodeep-scrub", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:21:25.505 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 4, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:21:25.506 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "jewel", 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:21:25.507 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:21:25.508 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:21:25.509 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:21:25.510 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:21:25.511 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:21:25.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 2.9900000095367432, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 2.9900000095367432, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.87999999523162842, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2.630000114440918, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2.630000114440918, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 1, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 1, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 1 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:21:25.514 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:21:25.515 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:21:25.516 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:21:25.517 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:21:25.518 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:21:25.519 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 7.690000057220459, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 7.690000057220459, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.12999999523162842, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 1, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 1, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 1 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:21:25.520 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:21:25.521 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:21:25.522 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "23", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 4, 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 23, 2024-04-23T21:21:25.523 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:21:25.524 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:21:25.525 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:21:25.526 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 1.7400000095367432, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 1.7400000095367432, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.87999999523162842, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 1, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 1, 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 1 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:25.527 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.528 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.529 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:21:25.531 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.533 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 0, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.534 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 14, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.535 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.538 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1285376165 2024-04-23T21:21:25.539 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/1285376165", 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/1285376165", 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/1285376165", 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/1285376165", 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "exists" 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:21:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.541 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:21:25.544 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:21:25.545 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.547 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.551 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.554 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.555 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.557 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:21:25.558 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.559 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.560 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.562 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:21:25.564 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.565 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.568 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.570 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:21:25.571 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.573 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:21:25.574 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.575 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:21:25.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:21:25.578 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.579 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.580 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:21:25.582 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:21:25.583 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:21:25.584 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [ 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.585 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 2, 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 1 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [], 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [], 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:21:25.586 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:25.587 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:25.587 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:25.504+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:25.589 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:21:23.792364+0000', 'last_up_change': '2024-04-23T21:21:16.757901+0000', 'last_in_change': '2024-04-23T21:20:59.765723+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.690000057220459, 'score_stable': 7.690000057220459, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 1, 'raw_score_stable': 1, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 23, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 14, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 1285376165}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 1285376165}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 1285376165}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 1285376165}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 1285376165}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 1285376165}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 1285376165}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 1285376165}]}, 'public_addr': '172.21.15.111:6817/1285376165', 'cluster_addr': '172.21.15.111:6819/1285376165', 'heartbeat_back_addr': '172.21.15.111:6823/1285376165', 'heartbeat_front_addr': '172.21.15.111:6821/1285376165', 'state': ['exists']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:21:25.589 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-23T21:21:25.590 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:25.590 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:25.590 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:25.590 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:25.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:25.600 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:25.601+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:25.612 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:25.612 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:25.630 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:25.630 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:25.649 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:25.649 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:25.665 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:25.665 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:25.682 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:25.683 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:25.700 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:25.700 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:25.701 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:25.702+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:25.715 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:25.715 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:25.718 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:25.718 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:25.721 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:25.724 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:25.798 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:21:25.802 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:25.802+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:25.871 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone2 from 0x52c3789 (overlap 0x2224b4) to 0x1b5090 2024-04-23T21:21:25.879 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:25.879 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:25.889 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:25.903 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:25.904+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:25.945 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-04-23T21:21:26.002 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.002 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.004 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:26.005+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:26.014 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:26.106 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:26.107+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:26.161 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.162 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.171 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:26.199 INFO:teuthology.orchestra.run.smithi111.stdout:258 writesame 0x311d03b thru 0x312bd03 data_size 0x1(0xecc9 bytes) 2024-04-23T21:21:26.206 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:26.206+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:26.233 INFO:teuthology.orchestra.run.smithi111.stdout:259 read 0x2318f33 thru 0x23242c0 (0xb38e bytes) 2024-04-23T21:21:26.233 INFO:teuthology.orchestra.run.smithi111.stdout:260 write 0xd7f28d thru 0xd82377 (0x30eb bytes) 2024-04-23T21:21:26.253 INFO:teuthology.orchestra.run.smithi111.stdout:262 writesame 0xbbf3122 thru 0xbc01d6d data_size 0x8e(0xec4c bytes) 2024-04-23T21:21:26.261 INFO:teuthology.orchestra.run.smithi111.stdout:263 write 0xe01b26a thru 0xe025c09 (0xa9a0 bytes) 2024-04-23T21:21:26.266 INFO:teuthology.orchestra.run.smithi111.stdout:264 punch from 0x3ec1a1b to 0x3ec321b, (0x1800 bytes) 2024-04-23T21:21:26.266 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xe9f5859 2024-04-23T21:21:26.266 INFO:teuthology.orchestra.run.smithi111.stdout:265 trunc from 0xe025c0a to 0xe9f5859 2024-04-23T21:21:26.271 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.290 INFO:teuthology.orchestra.run.smithi111.stdout:272 trunc from 0xe9f5859 to 0x8bac16 2024-04-23T21:21:26.292 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.293 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:26.307 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:26.307+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:26.330 INFO:teuthology.orchestra.run.smithi111.stdout:273 writesame 0x4853169 thru 0x485dff3 data_size 0x1(0xae8b bytes) 2024-04-23T21:21:26.330 INFO:teuthology.orchestra.run.smithi111.stdout:275 read 0x3042f6f thru 0x304e2a3 (0xb335 bytes) 2024-04-23T21:21:26.330 INFO:teuthology.orchestra.run.smithi111.stdout:276 read 0x1b13c8e thru 0x1b2281a (0xeb8d bytes) 2024-04-23T21:21:26.330 INFO:teuthology.orchestra.run.smithi111.stdout:278 trunc from 0x485dff4 to 0x3bf5a14 2024-04-23T21:21:26.347 INFO:teuthology.orchestra.run.smithi111.stdout:279 trunc from 0x3bf5a14 to 0xbcdb2c8 2024-04-23T21:21:26.355 INFO:teuthology.orchestra.run.smithi111.stdout:280 write 0x137d1e6 thru 0x13874c6 (0xa2e1 bytes) 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:281 punch from 0x8589a45 to 0x858b901, (0x1ebc bytes) 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:282 read 0x2749652 thru 0x274be34 (0x27e3 bytes) 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:283 read 0x7764473 thru 0x7766034 (0x1bc2 bytes) 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:285 punch from 0x5095c3c to 0x50a1d23, (0xc0e7 bytes) 2024-04-23T21:21:26.359 INFO:teuthology.orchestra.run.smithi111.stdout:286 trunc from 0xbcdb2c8 to 0xb75ab5e 2024-04-23T21:21:26.372 INFO:teuthology.orchestra.run.smithi111.stdout:287 write 0xcac8465 thru 0xcad67e7 (0xe383 bytes) 2024-04-23T21:21:26.390 INFO:teuthology.orchestra.run.smithi111.stdout:291 trunc from 0xcad67e8 to 0x400b55 2024-04-23T21:21:26.407 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:26.407+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:26.411 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.411 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:26.427 INFO:teuthology.orchestra.run.smithi111.stdout:292 write 0xa936111 thru 0xa936cc4 (0xbb4 bytes) 2024-04-23T21:21:26.430 INFO:teuthology.orchestra.run.smithi111.stdout:294 write 0x3139b63 thru 0x31464af (0xc94d bytes) 2024-04-23T21:21:26.433 INFO:teuthology.orchestra.run.smithi111.stdout:295 write 0x1bc908d thru 0x1bd821a (0xf18e bytes) 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:296 read 0x599daae thru 0x59a3469 (0x59bc bytes) 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:299 read 0x9493224 thru 0x949b4f1 (0x82ce bytes) 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:300 punch from 0x1adb649 to 0x1ae5a9d, (0xa454 bytes) 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:302 read 0x1f34724 thru 0x1f3d7b2 (0x908f bytes) 2024-04-23T21:21:26.436 INFO:teuthology.orchestra.run.smithi111.stdout:303 write 0x23421a9 thru 0x2350e32 (0xec8a bytes) 2024-04-23T21:21:26.442 INFO:teuthology.orchestra.run.smithi111.stdout:305 read 0x5f22445 thru 0x5f310a4 (0xec60 bytes) 2024-04-23T21:21:26.442 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.442 INFO:teuthology.orchestra.run.smithi111.stdout:308 punch from 0x32eb410 to 0x32f28d4, (0x74c4 bytes) 2024-04-23T21:21:26.448 INFO:teuthology.orchestra.run.smithi111.stdout:309 writesame 0xdd208af thru 0xdd2c3d7 data_size 0x1(0xbb29 bytes) 2024-04-23T21:21:26.448 INFO:teuthology.orchestra.run.smithi111.stdout:310 write 0xd1ab85e thru 0xd1b3425 (0x7bc8 bytes) 2024-04-23T21:21:26.451 INFO:teuthology.orchestra.run.smithi111.stdout:311 writesame 0xc4bdf6e thru 0xc4c21c0 data_size 0x1(0x4253 bytes) 2024-04-23T21:21:26.461 INFO:teuthology.orchestra.run.smithi111.stdout:312 read 0x70e1204 thru 0x70e3cb2 (0x2aaf bytes) 2024-04-23T21:21:26.461 INFO:teuthology.orchestra.run.smithi111.stdout:313 read 0x1e0c21e thru 0x1e15bb1 (0x9994 bytes) 2024-04-23T21:21:26.461 INFO:teuthology.orchestra.run.smithi111.stdout:314 read 0x510be4a thru 0x511b918 (0xfacf bytes) 2024-04-23T21:21:26.461 INFO:teuthology.orchestra.run.smithi111.stdout:315 read 0xcad7df3 thru 0xcadb024 (0x3232 bytes) 2024-04-23T21:21:26.461 INFO:teuthology.orchestra.run.smithi111.stdout:318 read 0x85247 thru 0x92dc9 (0xdb83 bytes) 2024-04-23T21:21:26.462 INFO:teuthology.orchestra.run.smithi111.stdout:319 read 0x3b71bae thru 0x3b7df26 (0xc379 bytes) 2024-04-23T21:21:26.462 INFO:teuthology.orchestra.run.smithi111.stdout:320 read 0x6c802c thru 0x6cbf68 (0x3f3d bytes) 2024-04-23T21:21:26.462 INFO:teuthology.orchestra.run.smithi111.stdout:322 write 0x7c4b1a6 thru 0x7c4d029 (0x1e84 bytes) 2024-04-23T21:21:26.464 INFO:teuthology.orchestra.run.smithi111.stdout:323 writesame 0xb27b42f thru 0xb280b32 data_size 0x2(0x5704 bytes) 2024-04-23T21:21:26.469 INFO:teuthology.orchestra.run.smithi111.stdout:324 read 0x821f095 thru 0x822d042 (0xdfae bytes) 2024-04-23T21:21:26.470 INFO:teuthology.orchestra.run.smithi111.stdout:325 write 0x77377a4 thru 0x773cc8c (0x54e9 bytes) 2024-04-23T21:21:26.473 INFO:teuthology.orchestra.run.smithi111.stdout:326 write 0x9c403d5 thru 0x9c42152 (0x1d7e bytes) 2024-04-23T21:21:26.475 INFO:teuthology.orchestra.run.smithi111.stdout:327 writesame 0x15bd5ce thru 0x15c495d data_size 0x2b(0x7390 bytes) 2024-04-23T21:21:26.479 INFO:teuthology.orchestra.run.smithi111.stdout:328 read 0x32d43c5 thru 0x32e089f (0xc4db bytes) 2024-04-23T21:21:26.479 INFO:teuthology.orchestra.run.smithi111.stdout:329 punch from 0x63c08fa to 0x63c9818, (0x8f1e bytes) 2024-04-23T21:21:26.479 INFO:teuthology.orchestra.run.smithi111.stdout:330 writesame 0xa2fe277 thru 0xa306c02 data_size 0x2(0x898c bytes) 2024-04-23T21:21:26.502 INFO:teuthology.orchestra.run.smithi111.stdout:331 trunc from 0xdd2c3d8 to 0x153120e 2024-04-23T21:21:26.508 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:26.508+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:26.514 INFO:teuthology.orchestra.run.smithi111.stdout:333 read 0x6f3bc0 thru 0x6f7290 (0x36d1 bytes) 2024-04-23T21:21:26.515 INFO:teuthology.orchestra.run.smithi111.stdout:334 punch from 0x143fc0f to 0x1447710, (0x7b01 bytes) 2024-04-23T21:21:26.515 INFO:teuthology.orchestra.run.smithi111.stdout:335 read 0x1359be8 thru 0x1363f5f (0xa378 bytes) 2024-04-23T21:21:26.529 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.529 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.534 INFO:teuthology.orchestra.run.smithi111.stdout:336 write 0xc04b9d1 thru 0xc05a890 (0xeec0 bytes) 2024-04-23T21:21:26.539 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:26.541 INFO:teuthology.orchestra.run.smithi111.stdout:337 write 0xca4dae3 thru 0xca5bb82 (0xe0a0 bytes) 2024-04-23T21:21:26.544 INFO:teuthology.orchestra.run.smithi111.stdout:339 read 0x8f9d31d thru 0x8fa9c02 (0xc8e6 bytes) 2024-04-23T21:21:26.544 INFO:teuthology.orchestra.run.smithi111.stdout:340 write 0x73ef80 thru 0x73f024 (0xa5 bytes) 2024-04-23T21:21:26.545 INFO:teuthology.orchestra.run.smithi111.stdout:341 read 0x1dbdddc thru 0x1dc0f16 (0x313b bytes) 2024-04-23T21:21:26.545 INFO:teuthology.orchestra.run.smithi111.stdout:343 read 0x337ad1d thru 0x337fc5f (0x4f43 bytes) 2024-04-23T21:21:26.545 INFO:teuthology.orchestra.run.smithi111.stdout:344 write 0x5f70fb9 thru 0x5f7e76b (0xd7b3 bytes) 2024-04-23T21:21:26.548 INFO:teuthology.orchestra.run.smithi111.stdout:345 punch from 0x2563896 to 0x2573291, (0xf9fb bytes) 2024-04-23T21:21:26.554 INFO:teuthology.orchestra.run.smithi111.stdout:346 write 0xcbaeb02 thru 0xcbb6c1b (0x811a bytes) 2024-04-23T21:21:26.557 INFO:teuthology.orchestra.run.smithi111.stdout:347 trunc from 0xcbb6c1c to 0xcef8cf2 2024-04-23T21:21:26.563 INFO:teuthology.orchestra.run.smithi111.stdout:350 read 0xc715e0b thru 0xc71fc75 (0x9e6b bytes) 2024-04-23T21:21:26.563 INFO:teuthology.orchestra.run.smithi111.stdout:353 punch from 0x601d0f9 to 0x602b3d8, (0xe2df bytes) 2024-04-23T21:21:26.563 INFO:teuthology.orchestra.run.smithi111.stdout:355 write 0x34c2c0b thru 0x34cbacb (0x8ec1 bytes) 2024-04-23T21:21:26.567 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.567 INFO:teuthology.orchestra.run.smithi111.stdout:357 read 0x7e2da90 thru 0x7e38d76 (0xb2e7 bytes) 2024-04-23T21:21:26.567 INFO:teuthology.orchestra.run.smithi111.stdout:358 read 0x440dc8 thru 0x44634f (0x5588 bytes) 2024-04-23T21:21:26.568 INFO:teuthology.orchestra.run.smithi111.stdout:359 writesame 0x467569 thru 0x473658 data_size 0xa8(0xc0f0 bytes) 2024-04-23T21:21:26.568 INFO:teuthology.orchestra.run.smithi111.stdout:360 write 0x252af06 thru 0x2537cba (0xcdb5 bytes) 2024-04-23T21:21:26.580 INFO:teuthology.orchestra.run.smithi111.stdout:361 writesame 0xe766e33 thru 0xe773b2c data_size 0x1(0xccfa bytes) 2024-04-23T21:21:26.580 INFO:teuthology.orchestra.run.smithi111.stdout:362 write 0x6e245bf thru 0x6e28d3b (0x477d bytes) 2024-04-23T21:21:26.582 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:26.583 INFO:teuthology.orchestra.run.smithi111.stdout:364 clone 4 order 25 su 8388608 sc 2 2024-04-23T21:21:26.609 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:26.610+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:26.643 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.644 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:26.710 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:26.710+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:26.795 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.795 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:26.850 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:26.851+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:26.881 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone3 from 0xe773b2d (overlap 0x400b55) to 0xccf11 2024-04-23T21:21:26.913 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:26.913+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:26.916 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:26.916 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:26.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:26.974 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-04-23T21:21:26.980 INFO:teuthology.orchestra.run.smithi111.stdout:365 read 0xd6b89c9 thru 0xd6bd0ef (0x4727 bytes) 2024-04-23T21:21:26.980 INFO:teuthology.orchestra.run.smithi111.stdout:366 clone 5 order 24 su 262144 sc 4 2024-04-23T21:21:27.014 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:27.014+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:27.068 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.068 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.077 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:27.114 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:27.115+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:27.215 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:27.216+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:27.218 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.219 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.228 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:27.315 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:27.316+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:27.368 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.369 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.377 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:27.416 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:27.417+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:27.518 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:27.518+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:27.520 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.520 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:27.619 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:27.619+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:27.672 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.672 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:27.722 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:27.724 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:27.725+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:27.782 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.783 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:27.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:21:27.850 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:27.850 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:27.851 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:27.850+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:27.851 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:27.872 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:27.872 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:27.891 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:27.891 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:27.908 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:27.908 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:27.917 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone4 from 0xe773b2d (overlap 0xe773b2d) to 0xb9295bd 2024-04-23T21:21:27.921 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:27.921+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:27.928 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:27.928 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:27.940 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:27.941 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:27.948 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:27.948 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:27.950 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:27.966 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:27.966 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:27.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:28.022 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:28.022+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.064 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:28.064 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:28.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:28.111 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-04-23T21:21:28.115 INFO:teuthology.orchestra.run.smithi111.stdout:368 punch from 0xd66ee11 to 0xd67bea2, (0xd091 bytes) 2024-04-23T21:21:28.122 INFO:teuthology.orchestra.run.smithi111.stdout:369 trunc from 0xe773b2d to 0xb0b2874 2024-04-23T21:21:28.123 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:28.124+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:28.150 INFO:teuthology.orchestra.run.smithi111.stdout:370 punch from 0x56d035a to 0x56d875a, (0x8400 bytes) 2024-04-23T21:21:28.150 INFO:teuthology.orchestra.run.smithi111.stdout:371 write 0x46ac320 thru 0x46b90a2 (0xcd83 bytes) 2024-04-23T21:21:28.157 INFO:teuthology.orchestra.run.smithi111.stdout:372 write 0xa48bb95 thru 0xa4903cc (0x4838 bytes) 2024-04-23T21:21:28.164 INFO:teuthology.orchestra.run.smithi111.stdout:374 trunc from 0xb0b2874 to 0x69dd97e 2024-04-23T21:21:28.222 INFO:teuthology.orchestra.run.smithi111.stdout:375 write 0x1c59d8d thru 0x1c5e283 (0x44f7 bytes) 2024-04-23T21:21:28.223 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:28.224+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:28.233 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.233 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:28.233 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:28.233 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:28.233 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:28.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:28.266 INFO:teuthology.orchestra.run.smithi111.stdout:376 write 0x90a3ec thru 0x910f17 (0x6b2c bytes) 2024-04-23T21:21:28.302 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.303 INFO:teuthology.orchestra.run.smithi111.stdout:379 read 0x286cae5 thru 0x287a47a (0xd996 bytes) 2024-04-23T21:21:28.317 INFO:teuthology.orchestra.run.smithi111.stdout:380 write 0xdaab6ae thru 0xdab1f71 (0x68c4 bytes) 2024-04-23T21:21:28.320 INFO:teuthology.orchestra.run.smithi111.stdout:382 writesame 0x5dcf192 thru 0x5dd58bf data_size 0x2f(0x672e bytes) 2024-04-23T21:21:28.324 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:28.325+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:28.336 INFO:teuthology.orchestra.run.smithi111.stdout:383 trunc from 0xdab1f72 to 0x1d95e4c 2024-04-23T21:21:28.361 INFO:teuthology.orchestra.run.smithi111.stdout:384 write 0x6e0567 thru 0x6eab30 (0xa5ca bytes) 2024-04-23T21:21:28.362 INFO:teuthology.orchestra.run.smithi111.stdout:385 punch from 0x1314e8a to 0x132040f, (0xb585 bytes) 2024-04-23T21:21:28.372 INFO:teuthology.orchestra.run.smithi111.stdout:386 trunc from 0x1d95e4c to 0xa34a4a5 2024-04-23T21:21:28.378 INFO:teuthology.orchestra.run.smithi111.stdout:388 writesame 0x7a69b63 thru 0x7a6b0d5 data_size 0x11(0x1573 bytes) 2024-04-23T21:21:28.381 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.381 INFO:teuthology.orchestra.run.smithi111.stdout:391 punch from 0x6e4e0f4 to 0x6e5cdb2, (0xecbe bytes) 2024-04-23T21:21:28.381 INFO:teuthology.orchestra.run.smithi111.stdout:392 read 0x7a4c013 thru 0x7a4c876 (0x864 bytes) 2024-04-23T21:21:28.381 INFO:teuthology.orchestra.run.smithi111.stdout:393 read 0x8ab77e8 thru 0x8abf895 (0x80ae bytes) 2024-04-23T21:21:28.381 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xed7a99c 2024-04-23T21:21:28.387 INFO:teuthology.orchestra.run.smithi111.stdout:394 trunc from 0xa34a4a5 to 0xed7a99c 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3269 3.18 e23/18 3:1ace6d32:::rbd_data.11183c92f5c.0000000000000024:head v 23'7, mlcod=23'5)", 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.8541336460000002, 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0030814610000000002, 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.388 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532948+0000", 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532949+0000", 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1999999999999998e-07 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532955+0000", 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7209999999999997e-06 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532966+0000", 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0848e-05 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.532975+0000", 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0040000000000005e-06 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.533041+0000", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5458999999999998e-05 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.533060+0000", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9726000000000001e-05 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.535997+0000", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0029370870000000001 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.536029+0000", 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1795999999999998e-05 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:3286 3.12 3:4e946cf6:::rbd_data.11183c92f5c.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,write 3889362~2790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.8338729589999998, 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028384349999999998, 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.393 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 3286 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553208+0000", 2024-04-23T21:21:28.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553209+0000", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6700000000000003e-07 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553214+0000", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7430000000000002e-06 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553224+0000", 2024-04-23T21:21:28.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0155e-05 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553229+0000", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3149999999999999e-06 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553272+0000", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2330999999999998e-05 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553353+0000", 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0822000000000006e-05 2024-04-23T21:21:28.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.553440+0000", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7670000000000006e-05 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.554029+0000", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00058887499999999999 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.555990+0000", 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0019609739999999999 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.556021+0000", 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0491999999999999e-05 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.556047+0000", 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6291e-05 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3365 3.3 e23/18 3:c75763a2:::rbd_data.11183c92f5c.0000000000000021:head v 23'9, mlcod=23'7)", 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.7356433629999999, 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0029539560000000002, 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651438+0000", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651439+0000", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9100000000000002e-07 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651446+0000", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2989999999999999e-06 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651456+0000", 2024-04-23T21:21:28.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5089999999999999e-06 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651460+0000", 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.729e-06 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651505+0000", 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4481000000000003e-05 2024-04-23T21:21:28.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.651518+0000", 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2707999999999999e-05 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.654357+0000", 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028396369999999999 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.654392+0000", 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4702000000000001e-05 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4904 3.12 3:49e172c6:::rbd_object_map.11181344e3fc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.181778+0000", 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.205302929, 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0038215969999999999, 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4904 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181778+0000", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181778+0000", 2024-04-23T21:21:28.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181779+0000", 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181782+0000", 2024-04-23T21:21:28.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.05e-06 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181787+0000", 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7169999999999999e-06 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.181791+0000", 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6310000000000002e-06 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182459+0000", 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00066738599999999996 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182526+0000", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7152999999999999e-05 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182760+0000", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00023389600000000001 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.183801+0000", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0010411629999999999 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185526+0000", 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0017250799999999999 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185566+0000", 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9823999999999997e-05 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185600+0000", 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4366000000000003e-05 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4905 3.12 3:49e172c6:::rbd_object_map.11181344e3fc:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.182630+0000", 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.204450934, 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0033380839999999998, 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4905 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182630+0000", 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182630+0000", 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182632+0000", 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1230000000000001e-06 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182636+0000", 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8250000000000004e-06 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182647+0000", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.182652+0000", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.183069+0000", 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041706399999999998 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.183122+0000", 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2843999999999998e-05 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.183339+0000", 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021628899999999999 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.184104+0000", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00076586799999999995 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185887+0000", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0017822109999999999 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185927+0000", 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9912e-05 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.185969+0000", 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1894000000000002e-05 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:4908 3.3 e26/18 3:c0665d5c:::rbd_data.11181344e3fc.0000000000000005:head v 26'17, mlcod=24'15)", 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.199281+0000", 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.18780029100000001, 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.018185692, 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.199281+0000", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.199281+0000", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.199283+0000", 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8700000000000001e-06 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.209998+0000", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010714703000000001 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.210025+0000", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7732000000000001e-05 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.210043+0000", 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7187999999999999e-05 2024-04-23T21:21:28.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.210122+0000", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9361000000000001e-05 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.210146+0000", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3719000000000001e-05 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.217391+0000", 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.007245647 2024-04-23T21:21:28.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.217467+0000", 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5471999999999998e-05 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4935 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 7445901~17655] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.230953+0000", 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.15612820599999999, 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.036728139999999999, 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4935 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.230953+0000", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.230953+0000", 2024-04-23T21:21:28.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.230955+0000", 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2709999999999999e-06 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.240880+0000", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0099246130000000005 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.240912+0000", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.188e-05 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.240924+0000", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2374e-05 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.241022+0000", 2024-04-23T21:21:28.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7553999999999995e-05 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.241177+0000", 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015482500000000001 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.241991+0000", 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00081438599999999995 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.258752+0000", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.016761300999999999 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.267603+0000", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0088509810000000008 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.267645+0000", 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1736000000000002e-05 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.267681+0000", 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6219e-05 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4953 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write 2401260~27436] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.273152+0000", 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.113929445, 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.031243445000000002, 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.426 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4953 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.273152+0000", 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.273152+0000", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.273154+0000", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1490000000000001e-06 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.277453+0000", 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0042986939999999996 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.277478+0000", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5168999999999999e-05 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.277490+0000", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2249000000000001e-05 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.277573+0000", 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3022000000000006e-05 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.277710+0000", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013688700000000001 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.278097+0000", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00038726500000000003 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.289785+0000", 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011687266999999999 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.304313+0000", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014528688 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.304360+0000", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6328000000000001e-05 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.304395+0000", 2024-04-23T21:21:28.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5726999999999999e-05 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4970 3.1 3:87bfb7b7:::rbd_data.11181344e3fc.0000000000000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.339396+0000", 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.047685168, 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0037304310000000002, 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4970 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339396+0000", 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339396+0000", 2024-04-23T21:21:28.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339397+0000", 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4799999999999998e-07 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339401+0000", 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8569999999999997e-06 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339410+0000", 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1009999999999998e-06 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339415+0000", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0710000000000002e-06 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339479+0000", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4166999999999997e-05 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339559+0000", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9604000000000006e-05 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339662+0000", 2024-04-23T21:21:28.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010382600000000001 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.340314+0000", 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00065165500000000001 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.343066+0000", 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0027518080000000001 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.343099+0000", 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3052999999999997e-05 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.343127+0000", 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7640999999999998e-05 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.439 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4971 3.1 3:80fa93cf:::rbd_data.11181344e3fc.0000000000000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.339429+0000", 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.047652100000000003, 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0066368340000000003, 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4971 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339429+0000", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339429+0000", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339430+0000", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2800000000000003e-07 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339445+0000", 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5104e-05 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339452+0000", 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0400000000000004e-06 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339456+0000", 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9430000000000002e-06 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339926+0000", 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000470493 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.339996+0000", 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9344999999999994e-05 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.340065+0000", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9846999999999997e-05 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.340590+0000", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00052481100000000003 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.346007+0000", 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0054173249999999997 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.346039+0000", 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1476000000000002e-05 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.346066+0000", 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7121999999999999e-05 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.445 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4974 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:head [stat,truncate 7864320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.347787+0000", 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.039294463000000002, 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050387890000000001, 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4974 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347787+0000", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347787+0000", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347787+0000", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4899999999999995e-07 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347792+0000", 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1289999999999998e-06 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347802+0000", 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9499999999999996e-06 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347813+0000", 2024-04-23T21:21:28.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1328000000000001e-05 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347860+0000", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7277999999999997e-05 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347939+0000", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8778999999999999e-05 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348083+0000", 2024-04-23T21:21:28.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014397600000000001 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352708+0000", 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046253259999999999 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352764+0000", 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5853000000000002e-05 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352796+0000", 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1608e-05 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352826+0000", 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0012999999999999e-05 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4975 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:head [stat,truncate 7864320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.347840+0000", 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.039241028999999997, 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0058798849999999996, 2024-04-23T21:21:28.451 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4975 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347840+0000", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347840+0000", 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347841+0000", 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347844+0000", 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.484e-06 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347852+0000", 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7659999999999998e-06 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347859+0000", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9820000000000002e-06 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.350348+0000", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0024887910000000002 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.350430+0000", 2024-04-23T21:21:28.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460999999999995e-05 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.350581+0000", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000150674 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352755+0000", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0021743270000000002 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353678+0000", 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00092211500000000004 2024-04-23T21:21:28.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353702+0000", 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4207e-05 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353720+0000", 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8459000000000001e-05 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.457 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4977 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.347913+0000", 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.039168847999999999, 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051586749999999997, 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4977 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347913+0000", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347913+0000", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347913+0000", 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3399999999999999e-07 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347916+0000", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4849999999999999e-06 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347931+0000", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5667000000000001e-05 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347937+0000", 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6099999999999997e-06 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352860+0000", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049234550000000002 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.352955+0000", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4258000000000005e-05 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353071+0000", 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000116666 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4980 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.347976+0000", 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.039105330000000001, 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0052860880000000004, 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4980 2024-04-23T21:21:28.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347976+0000", 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347976+0000", 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347977+0000", 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8400000000000005e-07 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347979+0000", 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3549999999999999e-06 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347985+0000", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1249999999999998e-06 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.347990+0000", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.254e-06 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353087+0000", 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005097178 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353163+0000", 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5709000000000003e-05 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353262+0000", 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8982999999999997e-05 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.466 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4983 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 1048576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.348017+0000", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.039064008999999997, 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0054623110000000001, 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4983 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348017+0000", 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348017+0000", 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348018+0000", 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348020+0000", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1560000000000002e-06 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348026+0000", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1160000000000004e-06 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348031+0000", 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6609999999999999e-06 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353295+0000", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0052639690000000003 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353380+0000", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5006999999999994e-05 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353480+0000", 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9906000000000002e-05 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4991 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.348190+0000", 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.038891292000000001, 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0055328490000000003, 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4991 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.471 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348190+0000", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348190+0000", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348190+0000", 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:28.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348192+0000", 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348198+0000", 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.818e-06 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348203+0000", 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.357e-06 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353512+0000", 2024-04-23T21:21:28.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0053095520000000004 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353596+0000", 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3795000000000007e-05 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353723+0000", 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012694900000000001 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4997 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 262144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.348802+0000", 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.038278885999999998, 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050728040000000002, 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.477 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4997 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348802+0000", 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348802+0000", 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348803+0000", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8000000000000005e-07 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348806+0000", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348813+0000", 2024-04-23T21:21:28.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0480000000000001e-06 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348819+0000", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0059999999999998e-06 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353760+0000", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049408860000000002 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353813+0000", 2024-04-23T21:21:28.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2643999999999999e-05 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353875+0000", 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2483999999999993e-05 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:5000 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 786432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.348850+0000", 2024-04-23T21:21:28.482 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.038231276000000002, 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051578379999999997, 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 5000 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348850+0000", 2024-04-23T21:21:28.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348850+0000", 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348851+0000", 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.06e-07 2024-04-23T21:21:28.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348853+0000", 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2589999999999999e-06 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348865+0000", 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2126999999999999e-05 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348871+0000", 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.604e-06 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353894+0000", 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050230880000000002 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.353954+0000", 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0464000000000002e-05 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354008+0000", 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3690000000000003e-05 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.487 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:5003 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 1310720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.348911+0000", 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.038170637, 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0052304810000000004, 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 5003 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348911+0000", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348911+0000", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348911+0000", 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6499999999999999e-07 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348914+0000", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.509e-06 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348920+0000", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7300000000000002e-06 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.348927+0000", 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4950000000000002e-06 2024-04-23T21:21:28.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354039+0000", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051120519999999997 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354088+0000", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8984999999999999e-05 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354141+0000", 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3145e-05 2024-04-23T21:21:28.491 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:5011 3.1b 3:db2742f1:::rbd_data.1118f7f89011.0000000000000003:6 [sparse-read 1835008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.349131+0000", 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.037950361000000002, 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051321099999999996, 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.492 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 5011 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349131+0000", 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349131+0000", 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349131+0000", 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349133+0000", 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7770000000000001e-06 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349139+0000", 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.699e-06 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.349144+0000", 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7690000000000004e-06 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354154+0000", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050104839999999999 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354211+0000", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6397999999999998e-05 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.354263+0000", 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2596999999999998e-05 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.496 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.497 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.497 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.497 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.497 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:395 read 0x22c6d91 thru 0x22cae38 (0x40a8 bytes) 2024-04-23T21:21:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:396 write 0xb214b30 thru 0xb21f334 (0xa805 bytes) 2024-04-23T21:21:28.498 INFO:teuthology.orchestra.run.smithi111.stdout:398 read 0x39ee278 thru 0x39f9e9f (0xbc28 bytes) 2024-04-23T21:21:28.499 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:28.501 INFO:teuthology.orchestra.run.smithi111.stdout:399 read 0x2e53836 thru 0x2e54778 (0xf43 bytes) 2024-04-23T21:21:28.501 INFO:teuthology.orchestra.run.smithi111.stdout:402 write 0x62352c1 thru 0x623cda5 (0x7ae5 bytes) 2024-04-23T21:21:28.502 INFO:teuthology.orchestra.run.smithi111.stdout:403 trunc from 0xed7a99c to 0x15c25fa 2024-04-23T21:21:28.502 INFO:teuthology.orchestra.run.smithi111.stdout:406 read 0x104c55e thru 0x104fcf7 (0x379a bytes) 2024-04-23T21:21:28.502 INFO:teuthology.orchestra.run.smithi111.stdout:408 read 0xe2e9e1 thru 0xe37eb2 (0x94d2 bytes) 2024-04-23T21:21:28.502 INFO:teuthology.orchestra.run.smithi111.stdout:409 punch from 0x71c4e8 to 0x7258b5, (0x93cd bytes) 2024-04-23T21:21:28.502 INFO:teuthology.orchestra.run.smithi111.stdout:410 punch from 0x947f6a to 0x94fc27, (0x7cbd bytes) 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:411 write 0x78bf0c3 thru 0x78c518b (0x60c9 bytes) 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:414 read 0xef8854 thru 0xef97c7 (0xf74 bytes) 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:415 trunc from 0x78c518c to 0x1b36aeb 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:416 trunc from 0x1b36aeb to 0x33106c0 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:417 trunc from 0x33106c0 to 0xc511237 2024-04-23T21:21:28.503 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.504 INFO:teuthology.orchestra.run.smithi111.stdout:420 read 0x520fe91 thru 0x5210542 (0x6b2 bytes) 2024-04-23T21:21:28.504 INFO:teuthology.orchestra.run.smithi111.stdout:422 read 0x5cd4502 thru 0x5cd8057 (0x3b56 bytes) 2024-04-23T21:21:28.504 INFO:teuthology.orchestra.run.smithi111.stdout:423 punch from 0x22a0dab to 0x22ad1a7, (0xc3fc bytes) 2024-04-23T21:21:28.504 INFO:teuthology.orchestra.run.smithi111.stdout:425 read 0x1409f6f thru 0x14131e7 (0x9279 bytes) 2024-04-23T21:21:28.504 INFO:teuthology.orchestra.run.smithi111.stdout:426 read 0x5382644 thru 0x538b40a (0x8dc7 bytes) 2024-04-23T21:21:28.504 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:28.498+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:28.505 INFO:teuthology.orchestra.run.smithi111.stdout:427 writesame 0xcbc2f2e thru 0xcbc4987 data_size 0x1(0x1a5a bytes) 2024-04-23T21:21:28.516 INFO:teuthology.orchestra.run.smithi111.stdout:428 trunc from 0xcbc4988 to 0x800d60 2024-04-23T21:21:28.528 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.528 INFO:teuthology.orchestra.run.smithi111.stdout:431 punch from 0x664f2 to 0x6988e, (0x339c bytes) 2024-04-23T21:21:28.528 INFO:teuthology.orchestra.run.smithi111.stdout:432 punch from 0x24b60b to 0x24fc01, (0x45f6 bytes) 2024-04-23T21:21:28.528 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.536 INFO:teuthology.orchestra.run.smithi111.stdout:435 write 0x26e1458 thru 0x26f00e6 (0xec8f bytes) 2024-04-23T21:21:28.545 INFO:teuthology.orchestra.run.smithi111.stdout:436 writesame 0x40286de thru 0x403249c data_size 0x15(0x9dbf bytes) 2024-04-23T21:21:28.554 INFO:teuthology.orchestra.run.smithi111.stdout:437 writesame 0x603d5f1 thru 0x603d915 data_size 0x7(0x325 bytes) 2024-04-23T21:21:28.554 INFO:teuthology.orchestra.run.smithi111.stdout:438 writesame 0x427dc68 thru 0x4283953 data_size 0x26(0x5cec bytes) 2024-04-23T21:21:28.560 INFO:teuthology.orchestra.run.smithi111.stdout:439 read 0x311cc5f thru 0x312278b (0x5b2d bytes) 2024-04-23T21:21:28.560 INFO:teuthology.orchestra.run.smithi111.stdout:440 writesame 0x819bf4 thru 0x81e7ab data_size 0x4(0x4bb8 bytes) 2024-04-23T21:21:28.570 INFO:teuthology.orchestra.run.smithi111.stdout:441 trunc from 0x603d916 to 0xb9e4e5d 2024-04-23T21:21:28.576 INFO:teuthology.orchestra.run.smithi111.stdout:443 write 0xa31d0ae thru 0xa327c04 (0xab57 bytes) 2024-04-23T21:21:28.578 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.579 INFO:teuthology.orchestra.run.smithi111.stdout:446 read 0x47f0edc thru 0x47f74c5 (0x65ea bytes) 2024-04-23T21:21:28.579 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.579 INFO:teuthology.orchestra.run.smithi111.stdout:448 write 0xa790d1b thru 0xa796523 (0x5809 bytes) 2024-04-23T21:21:28.581 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:28.584 INFO:teuthology.orchestra.run.smithi111.stdout:450 trunc from 0xb9e4e5d to 0x96c0c3b 2024-04-23T21:21:28.591 INFO:teuthology.orchestra.run.smithi111.stdout:451 writesame 0x43bb494 thru 0x43c9f4b data_size 0x94(0xeab8 bytes) 2024-04-23T21:21:28.595 INFO:teuthology.orchestra.run.smithi111.stdout:453 trunc from 0x96c0c3b to 0x96659b9 2024-04-23T21:21:28.598 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:28.599+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:28.614 INFO:teuthology.orchestra.run.smithi111.stdout:454 write 0xea93d3e thru 0xea94755 (0xa18 bytes) 2024-04-23T21:21:28.627 INFO:teuthology.orchestra.run.smithi111.stdout:455 trunc from 0xea94756 to 0x7cfad66 2024-04-23T21:21:28.639 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.639 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:28.639 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:28.639 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:28.649 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:28.652 INFO:teuthology.orchestra.run.smithi111.stdout:456 trunc from 0x7cfad66 to 0xd0f022d 2024-04-23T21:21:28.657 INFO:teuthology.orchestra.run.smithi111.stdout:457 read 0x1986a03 thru 0x198fbcf (0x91cd bytes) 2024-04-23T21:21:28.658 INFO:teuthology.orchestra.run.smithi111.stdout:458 clone 6 order 22 su 32768 sc 10 2024-04-23T21:21:28.699 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:28.700+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:28.754 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.754 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:28.754 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:28.754 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:28.754 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:28.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:28.804 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:28.804+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:28.820 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:21:28.903 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:28.903+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:28.911 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone5 from 0xd0f022d (overlap 0x800d60) to 0x0 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:1487 3.9 e22/18 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:head v 22'13, mlcod=22'11)", 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.414161+0000", 2024-04-23T21:21:28.920 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.5054549870000002, 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00058172699999999998, 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414161+0000", 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414161+0000", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414162+0000", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8700000000000004e-07 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414169+0000", 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3529999999999997e-06 2024-04-23T21:21:28.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414181+0000", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1377e-05 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414187+0000", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1290000000000001e-06 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414247+0000", 2024-04-23T21:21:28.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0068999999999997e-05 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414272+0000", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5211e-05 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414708+0000", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00043546600000000002 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.414743+0000", 2024-04-23T21:21:28.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5135000000000002e-05 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:1514 3.9 e22/18 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:head v 22'14, mlcod=22'12)", 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.452611+0000", 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.4670054019999998, 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068577799999999995, 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452611+0000", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452611+0000", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452612+0000", 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.206e-06 2024-04-23T21:21:28.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452626+0000", 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4333e-05 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452645+0000", 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9120999999999999e-05 2024-04-23T21:21:28.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452656+0000", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0634e-05 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452718+0000", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2174000000000005e-05 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.452738+0000", 2024-04-23T21:21:28.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0152999999999998e-05 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.453249+0000", 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00051084000000000004 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.453296+0000", 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7317000000000001e-05 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1609 3.0 3:017a9918:::rbd_object_map.1118a1ef21e9.0000000000000003:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.830342+0000", 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0892737940000004, 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002334179, 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1609 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.930 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830342+0000", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830342+0000", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830344+0000", 2024-04-23T21:21:28.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9570000000000001e-06 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830352+0000", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4429999999999997e-06 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830366+0000", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4443999999999999e-05 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830373+0000", 2024-04-23T21:21:28.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7819999999999996e-06 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830423+0000", 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9650000000000001e-05 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op must wait for map", 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.830428+0000", 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4060000000000004e-06 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.832519+0000", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0020914060000000001 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.832567+0000", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7744000000000002e-05 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.832676+0000", 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109347 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.934 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1615 3.0 3:04479396:::rbd_id.image_client.0-clone2:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.836173+0000", 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0834430690000003, 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00083079699999999998, 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.935 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1615 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836173+0000", 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836173+0000", 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836174+0000", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8599999999999997e-07 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836179+0000", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8409999999999999e-06 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836189+0000", 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0457e-05 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836194+0000", 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2270000000000001e-06 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836232+0000", 2024-04-23T21:21:28.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7517e-05 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836310+0000", 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8269000000000006e-05 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836929+0000", 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00061864899999999998 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.836963+0000", 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4666000000000003e-05 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.837004+0000", 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0385e-05 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1620 3.0 3:04479396:::rbd_id.image_client.0-clone2:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.842598+0000", 2024-04-23T21:21:28.940 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0770183859999998, 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00037677199999999999, 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1620 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842598+0000", 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842598+0000", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842598+0000", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4000000000000002e-07 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842601+0000", 2024-04-23T21:21:28.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8250000000000001e-06 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842607+0000", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4339999999999997e-06 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842612+0000", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.369e-06 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842645+0000", 2024-04-23T21:21:28.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3296000000000001e-05 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842696+0000", 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0643999999999998e-05 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.842974+0000", 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00027876399999999998 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1655 3.0 3:04479396:::rbd_id.image_client.0-clone2:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:23.985063+0000", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.934553213, 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00037889299999999999, 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1655 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985063+0000", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985063+0000", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985063+0000", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0399999999999996e-07 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985066+0000", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0400000000000001e-06 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985073+0000", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.686e-06 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985081+0000", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4540000000000001e-06 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985128+0000", 2024-04-23T21:21:28.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7465e-05 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985212+0000", 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4308999999999995e-05 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:23.985442+0000", 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000229435 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1689 3.0 3:017a9918:::rbd_object_map.1118a1ef21e9.0000000000000003:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.003942+0000", 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.9156743020000002, 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00038538100000000002, 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.949 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1689 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003942+0000", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003942+0000", 2024-04-23T21:21:28.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003943+0000", 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6700000000000003e-07 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003947+0000", 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2810000000000002e-06 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003957+0000", 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9189999999999994e-06 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.003962+0000", 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0699999999999997e-06 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.004011+0000", 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9397000000000002e-05 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.004094+0000", 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3160999999999999e-05 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.004327+0000", 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00023278599999999999 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1724 3.4 3:2791df5c:::rbd_data.1118a1ef21e9.0000000000000013:head [sparse-read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.953 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.066778+0000", 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.8528377970000003, 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00026610400000000003, 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1724 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066778+0000", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066778+0000", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066780+0000", 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:21:28.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066793+0000", 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2578999999999999e-05 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066816+0000", 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2954000000000001e-05 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066826+0000", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0516999999999999e-05 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066936+0000", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109158 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067044+0000", 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000108727 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.957 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1726 3.4 3:2240345b:::rbd_data.1118a1ef21e9.0000000000000015:head [sparse-read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.066863+0000", 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.852752637, 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000279385, 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1726 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066863+0000", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066863+0000", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066864+0000", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0699999999999997e-07 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066868+0000", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6109999999999998e-06 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066878+0000", 2024-04-23T21:21:28.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0200000000000001e-05 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.066884+0000", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1979999999999998e-06 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067069+0000", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001853 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067143+0000", 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3468999999999997e-05 2024-04-23T21:21:28.961 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1733 3.4 3:21ef47ea:::rbd_data.1118a1ef21e9.000000000000001c:head [sparse-read 0~55845] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.067413+0000", 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.852202707, 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000309821, 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.962 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1733 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067413+0000", 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067413+0000", 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067414+0000", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7899999999999997e-07 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067419+0000", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6140000000000002e-06 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067438+0000", 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9309000000000001e-05 2024-04-23T21:21:28.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067446+0000", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8469999999999996e-06 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067494+0000", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8056000000000002e-05 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067564+0000", 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0148999999999995e-05 2024-04-23T21:21:28.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067723+0000", 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015906699999999999 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:1748 3.0 3:074685b7:::rbd_data.1118a1ef21e9.000000000000002b:head [sparse-read 0~40614] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.067745+0000", 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.8518710929999997, 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00036216200000000002, 2024-04-23T21:21:28.966 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 1748 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067745+0000", 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067745+0000", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067746+0000", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1999999999999999e-07 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067752+0000", 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6069999999999996e-06 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067764+0000", 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1983e-05 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067771+0000", 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5100000000000004e-06 2024-04-23T21:21:28.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067846+0000", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4819000000000003e-05 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.067922+0000", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6268999999999998e-05 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.068107+0000", 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018525400000000001 2024-04-23T21:21:28.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:2161 3.9 e23/18 3:9641a9b6:::rbd_data.11183c92f5c.000000000000000b:head v 23'15, mlcod=22'13)", 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.319075+0000", 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.6005408699999997, 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000872534, 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319075+0000", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319075+0000", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319075+0000", 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7700000000000001e-07 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319116+0000", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0417000000000003e-05 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319125+0000", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5310000000000007e-06 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319131+0000", 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1290000000000001e-06 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319174+0000", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2537000000000003e-05 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319189+0000", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4725e-05 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319898+0000", 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00070932999999999999 2024-04-23T21:21:28.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.319948+0000", 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9688000000000003e-05 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:2569 3.9 e23/18 3:9641a9b6:::rbd_data.11183c92f5c.000000000000000b:head v 23'16, mlcod=22'14)", 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.397619+0000", 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.5219975100000003, 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00050970099999999999, 2024-04-23T21:21:28.975 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397619+0000", 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397619+0000", 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397620+0000", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.353e-06 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397639+0000", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9465000000000001e-05 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397654+0000", 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4225e-05 2024-04-23T21:21:28.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397660+0000", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5760000000000002e-06 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397705+0000", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5046000000000002e-05 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.397721+0000", 2024-04-23T21:21:28.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5345e-05 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.398094+0000", 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00037369299999999997 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.398128+0000", 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3998000000000002e-05 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:3530 3.9 e23/18 3:9641a9b6:::rbd_data.11183c92f5c.000000000000000b:head v 23'17, mlcod=23'15)", 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.831045+0000", 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.088571344, 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00069814900000000001, 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831045+0000", 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831045+0000", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831046+0000", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1060000000000001e-06 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831056+0000", 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0026e-05 2024-04-23T21:21:28.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831069+0000", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3648999999999999e-05 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831076+0000", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8360000000000003e-06 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831131+0000", 2024-04-23T21:21:28.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4500999999999999e-05 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831153+0000", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2682999999999999e-05 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831711+0000", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000557104 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.831743+0000", 2024-04-23T21:21:28.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2243999999999998e-05 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4653 3.0 3:017a9918:::rbd_object_map.1118a1ef21e9.0000000000000003:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:27.881612+0000", 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.03800424, 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018494410000000001, 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4653 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881612+0000", 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881612+0000", 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881614+0000", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.131e-06 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881625+0000", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1236999999999999e-05 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881645+0000", 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9559999999999999e-05 2024-04-23T21:21:28.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881655+0000", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0012e-05 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881709+0000", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4694999999999997e-05 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op must wait for map", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.881713+0000", 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0269999999999997e-06 2024-04-23T21:21:28.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.883144+0000", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0014302729999999999 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.883233+0000", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9192999999999998e-05 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.883461+0000", 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00022831300000000001 2024-04-23T21:21:28.988 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4695 3.0 3:07e8b6ed:::rbd_data.11189b6d8e9d.0000000000000006:head [stat,delete] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:27.919145+0000", 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.0004713460000001, 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00089076100000000001, 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.989 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4695 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919145+0000", 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919145+0000", 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919146+0000", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.423e-06 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919152+0000", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919165+0000", 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2445e-05 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919172+0000", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6000000000000003e-06 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919226+0000", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4843999999999999e-05 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.919316+0000", 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9456999999999994e-05 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920035+0000", 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00071965099999999997 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4697 3.0 3:05b80c9a:::rbd_data.1118f7f89011.0000000000000060:6 [sparse-read 0~61120] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:27.920158+0000", 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.99945799999999996, 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0025369920000000001, 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.993 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4697 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920158+0000", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920158+0000", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920159+0000", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7800000000000001e-07 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920164+0000", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:28.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920176+0000", 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1861999999999999e-05 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920183+0000", 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1339999999999998e-06 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920247+0000", 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3981999999999998e-05 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920342+0000", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4677000000000002e-05 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922695+0000", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002353417 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4700 3.0 3:05b80c9a:::rbd_data.1118f7f89011.0000000000000060:6 [sparse-read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:27.920326+0000", 2024-04-23T21:21:28.997 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.99929004600000004, 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0025410150000000002, 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4700 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920326+0000", 2024-04-23T21:21:28.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920326+0000", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920327+0000", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7400000000000002e-07 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920332+0000", 2024-04-23T21:21:28.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7559999999999999e-06 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920343+0000", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1651000000000001e-05 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.920351+0000", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6179999999999997e-06 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922725+0000", 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0023745350000000001 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922787+0000", 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2001000000000005e-05 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922867+0000", 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9579999999999994e-05 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4703 3.0 3:07e8b6ed:::rbd_data.11189b6d8e9d.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,delete] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:29.001 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:27.922899+0000", 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.99671727600000004, 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0037664830000000002, 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4703 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922899+0000", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922899+0000", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922900+0000", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1429999999999999e-06 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922905+0000", 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4970000000000001e-06 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922918+0000", 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2159e-05 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922924+0000", 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9310000000000002e-06 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.922982+0000", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7250000000000002e-05 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.923070+0000", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7906999999999997e-05 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.926553+0000", 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.003483734 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.926596+0000", 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2509999999999998e-05 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:27.926665+0000", 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9351999999999996e-05 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:4793 3.0 3:017a9918:::rbd_object_map.1118a1ef21e9.0000000000000003:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:29.006 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:28.081420+0000", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.83819601300000002, 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00026676, 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 4793 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081420+0000", 2024-04-23T21:21:29.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081420+0000", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081422+0000", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4759999999999999e-06 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081428+0000", 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.64e-06 2024-04-23T21:21:29.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081448+0000", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9636e-05 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081455+0000", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8889999999999996e-06 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081505+0000", 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0111999999999997e-05 2024-04-23T21:21:29.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081561+0000", 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6474999999999997e-05 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:28.081687+0000", 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000125532 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.010 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:29.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:29.012 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:29.012+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:29.021 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-04-23T21:21:29.112 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:29.114+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:29.129 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:29.129 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:29.129 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:29.129 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:29.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:29.224 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:29.224+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:29.262 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.262 INFO:teuthology.orchestra.run.smithi111.stdout:460 writesame 0xc641295 thru 0xc650e68 data_size 0x4(0xfbd4 bytes) 2024-04-23T21:21:29.274 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.274 INFO:teuthology.orchestra.run.smithi111.stdout:463 write 0x164b927 thru 0x164c1d0 (0x8aa bytes) 2024-04-23T21:21:29.291 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:29.292 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:29.292 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:29.292 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:29.292 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:29.301 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:29.304 INFO:teuthology.orchestra.run.smithi111.stdout:464 write 0xe99f814 thru 0xe9a5ee0 (0x66cd bytes) 2024-04-23T21:21:29.313 INFO:teuthology.orchestra.run.smithi111.stdout:465 read 0x5c84d7d thru 0x5c85784 (0xa08 bytes) 2024-04-23T21:21:29.314 INFO:teuthology.orchestra.run.smithi111.stdout:467 punch from 0x2f8b240 to 0x2f9ac21, (0xf9e1 bytes) 2024-04-23T21:21:29.315 INFO:teuthology.orchestra.run.smithi111.stdout:468 read 0x3263f4c thru 0x326d067 (0x911c bytes) 2024-04-23T21:21:29.315 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.315 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:29.315+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:29.318 INFO:teuthology.orchestra.run.smithi111.stdout:470 trunc from 0xe9a5ee1 to 0x9304e58 2024-04-23T21:21:29.367 INFO:teuthology.orchestra.run.smithi111.stdout:471 writesame 0xeded836 thru 0xedf8e25 data_size 0x52(0xb5f0 bytes) 2024-04-23T21:21:29.367 INFO:teuthology.orchestra.run.smithi111.stdout:472 writesame 0x3381ed2 thru 0x338cb37 data_size 0x1(0xac66 bytes) 2024-04-23T21:21:29.395 INFO:teuthology.orchestra.run.smithi111.stdout:475 punch from 0x8c1c80a to 0x8c22984, (0x617a bytes) 2024-04-23T21:21:29.395 INFO:teuthology.orchestra.run.smithi111.stdout:476 punch from 0x6be0684 to 0x6be83e0, (0x7d5c bytes) 2024-04-23T21:21:29.412 INFO:teuthology.orchestra.run.smithi111.stdout:477 trunc from 0xedf8e26 to 0x4eddcf 2024-04-23T21:21:29.415 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:29.416+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:29.440 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:29.440 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:29.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2747 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 2916352~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403079+0000", 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0361283050000001, 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020690369, 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2747 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.441 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403079+0000", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403079+0000", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403079+0000", 2024-04-23T21:21:29.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.11e-07 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403083+0000", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.647e-06 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403086+0000", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.999e-06 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403098+0000", 2024-04-23T21:21:29.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605e-05 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423669+0000", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020571005 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423705+0000", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6091999999999999e-05 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423769+0000", 2024-04-23T21:21:29.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3810000000000001e-05 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2748 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 2981888~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403104+0000", 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0361032349999997, 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020796124999999999, 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2748 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403104+0000", 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403104+0000", 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403104+0000", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4200000000000002e-07 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403105+0000", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3659999999999999e-06 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403108+0000", 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.101e-06 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403111+0000", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6019999999999998e-06 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423793+0000", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020681511999999999 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423829+0000", 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6185e-05 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423900+0000", 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1117000000000004e-05 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2749 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3047424~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403115+0000", 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0360918569999997, 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020966348999999999, 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2749 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403115+0000", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403115+0000", 2024-04-23T21:21:29.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403115+0000", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7799999999999997e-07 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403117+0000", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.455e-06 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403128+0000", 2024-04-23T21:21:29.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1347e-05 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403139+0000", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1268e-05 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423924+0000", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020784435 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.423978+0000", 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3792000000000003e-05 2024-04-23T21:21:29.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424081+0000", 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000103774 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2750 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3112960~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403146+0000", 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0360611180000001, 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021124647999999999, 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2750 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403146+0000", 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403146+0000", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403146+0000", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7000000000000001e-07 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403148+0000", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.108e-06 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403152+0000", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1540000000000004e-06 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403155+0000", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6520000000000002e-06 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424127+0000", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020972322000000002 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424189+0000", 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1366999999999997e-05 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424270+0000", 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1774999999999995e-05 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2751 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3178496~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403158+0000", 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0360487699999998, 2024-04-23T21:21:29.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021270173, 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2751 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403158+0000", 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403158+0000", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403158+0000", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.05e-07 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403159+0000", 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2500000000000004e-07 2024-04-23T21:21:29.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403162+0000", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.88e-06 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403164+0000", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424299+0000", 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021134156000000001 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424342+0000", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3634000000000003e-05 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424428+0000", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6102999999999999e-05 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2752 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3244032~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403167+0000", 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.036039819, 2024-04-23T21:21:29.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021458758000000001, 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2752 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403167+0000", 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403167+0000", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403167+0000", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403168+0000", 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5099999999999998e-07 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403176+0000", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.238e-06 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403178+0000", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.587e-06 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424465+0000", 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021286637000000001 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424508+0000", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3346000000000001e-05 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424626+0000", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011779 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2753 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3309568~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403181+0000", 2024-04-23T21:21:29.465 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0360254539999998, 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021651449, 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2753 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403181+0000", 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403181+0000", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403182+0000", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.16e-07 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403183+0000", 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.068e-06 2024-04-23T21:21:29.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403186+0000", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9289999999999998e-06 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403189+0000", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7930000000000002e-06 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424663+0000", 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021474307000000002 2024-04-23T21:21:29.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424722+0000", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9058000000000003e-05 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424833+0000", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000111078 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2754 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3375104~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403198+0000", 2024-04-23T21:21:29.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0360093260000003, 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021833587000000002, 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2754 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403198+0000", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403198+0000", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403198+0000", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.17e-07 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403199+0000", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1850000000000001e-06 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403202+0000", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0429999999999999e-06 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403207+0000", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2050000000000001e-06 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424861+0000", 2024-04-23T21:21:29.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021653552999999999 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.424909+0000", 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8093999999999997e-05 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425031+0000", 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012229 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.473 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2755 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3440640~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403210+0000", 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359964750000001, 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022068859, 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2755 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403210+0000", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403210+0000", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403211+0000", 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1400000000000001e-07 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403212+0000", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0950000000000001e-06 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403216+0000", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7809999999999999e-06 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403229+0000", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3179e-05 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425072+0000", 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.021843544999999999 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425148+0000", 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5252000000000004e-05 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425279+0000", 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000131793 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2756 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3506176~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403236+0000", 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359708589999999, 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022360708, 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2756 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403236+0000", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403236+0000", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403236+0000", 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3799999999999999e-07 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403238+0000", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3200000000000001e-06 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403242+0000", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0130000000000004e-06 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403245+0000", 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4790000000000001e-06 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425337+0000", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022092100999999999 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425439+0000", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101854 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425597+0000", 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000157703 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2757 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3571712~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403248+0000", 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359587980000002, 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022532008999999999, 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2757 2024-04-23T21:21:29.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403248+0000", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403248+0000", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403248+0000", 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:21:29.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403249+0000", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0529999999999999e-06 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403252+0000", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9100000000000001e-06 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403255+0000", 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8629999999999999e-06 2024-04-23T21:21:29.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425636+0000", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022381000000000002 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425688+0000", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2207999999999999e-05 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425780+0000", 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1765e-05 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.485 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2758 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3637248~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403260+0000", 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359469319999999, 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022682117000000002, 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2758 2024-04-23T21:21:29.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403260+0000", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403260+0000", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403260+0000", 2024-04-23T21:21:29.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9999999999999999e-07 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403261+0000", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.052e-06 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403264+0000", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7599999999999998e-06 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403266+0000", 2024-04-23T21:21:29.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.311e-06 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425799+0000", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022532970999999999 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425853+0000", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4054000000000001e-05 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425942+0000", 2024-04-23T21:21:29.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8769000000000003e-05 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2759 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3702784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403279+0000", 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359274510000001, 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022808782999999999, 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2759 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403279+0000", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403279+0000", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403280+0000", 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3200000000000001e-07 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403281+0000", 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3370000000000001e-06 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403285+0000", 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3950000000000003e-06 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403288+0000", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0189999999999998e-06 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.425976+0000", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022687565 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426024+0000", 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7834000000000003e-05 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426088+0000", 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4400999999999995e-05 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2760 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3768320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403297+0000", 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0359102069999997, 2024-04-23T21:21:29.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022969191, 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2760 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403297+0000", 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403297+0000", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403297+0000", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2499999999999999e-07 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403298+0000", 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.375e-06 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403302+0000", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7610000000000001e-06 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403306+0000", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.844e-06 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426130+0000", 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022823683000000001 2024-04-23T21:21:29.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426187+0000", 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7018999999999997e-05 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426266+0000", 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9283999999999995e-05 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2761 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3833856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403311+0000", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358956509999997, 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023093430000000002, 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2761 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403311+0000", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403311+0000", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403312+0000", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3999999999999997e-07 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403315+0000", 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9909999999999998e-06 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403319+0000", 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2490000000000003e-06 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403321+0000", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6120000000000001e-06 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426293+0000", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.022971437000000001 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426340+0000", 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6814000000000003e-05 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426405+0000", 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4987000000000006e-05 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2762 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3899392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403325+0000", 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358823389999996, 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023242564, 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.503 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2762 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403325+0000", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403325+0000", 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403325+0000", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.22e-07 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403326+0000", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3500000000000005e-07 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403329+0000", 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7549999999999999e-06 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403350+0000", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1053e-05 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426445+0000", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023095683999999998 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426496+0000", 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1224000000000003e-05 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426567+0000", 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0691000000000005e-05 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2763 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 3964928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403355+0000", 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358519580000003, 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023360970000000002, 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.507 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2763 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403355+0000", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403355+0000", 2024-04-23T21:21:29.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403355+0000", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.65e-07 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403357+0000", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.313e-06 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403360+0000", 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8789999999999996e-06 2024-04-23T21:21:29.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403363+0000", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4839999999999998e-06 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426598+0000", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023235380999999999 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426650+0000", 2024-04-23T21:21:29.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1706999999999998e-05 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426716+0000", 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5940999999999998e-05 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2764 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 4030464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403366+0000", 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358410999999998, 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023491688, 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.511 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2764 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403366+0000", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403366+0000", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403366+0000", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0800000000000001e-07 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403367+0000", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0210000000000001e-06 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403370+0000", 2024-04-23T21:21:29.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7669999999999999e-06 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403372+0000", 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.34e-06 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426749+0000", 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023376946999999999 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426793+0000", 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3906999999999998e-05 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426858+0000", 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4498000000000005e-05 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2765 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 4096000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403375+0000", 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358321110000004, 2024-04-23T21:21:29.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023621629000000002, 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2765 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403375+0000", 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403375+0000", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403375+0000", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0699999999999999e-07 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403376+0000", 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0419999999999999e-06 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403379+0000", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8009999999999999e-06 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403381+0000", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426887+0000", 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023506091 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426936+0000", 2024-04-23T21:21:29.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8471999999999999e-05 2024-04-23T21:21:29.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.426996+0000", 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0696000000000003e-05 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:2766 3.19 3:9e246c11:::rbd_data.1118c9f76dd8.0000000000000000:4 [read 4161536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:24.403384+0000", 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 5.0358229330000004, 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023737508000000001, 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:29.522 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 2766 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403384+0000", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403384+0000", 2024-04-23T21:21:29.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403384+0000", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403385+0000", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3099999999999996e-07 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403388+0000", 2024-04-23T21:21:29.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7499999999999999e-06 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.403390+0000", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.427023+0000", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023632400000000001 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.427065+0000", 2024-04-23T21:21:29.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2429000000000003e-05 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:24.427122+0000", 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6468000000000002e-05 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:29.526 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:29.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:29.528 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:29.528+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:29.538 INFO:teuthology.orchestra.run.smithi111.stdout:480 read 0x45a4e6 thru 0x468738 (0xe253 bytes) 2024-04-23T21:21:29.557 INFO:teuthology.orchestra.run.smithi111.stdout:481 write 0xaaf1ee1 thru 0xaaf398b (0x1aab bytes) 2024-04-23T21:21:29.562 INFO:teuthology.orchestra.run.smithi111.stdout:482 read 0x7c375c7 thru 0x7c43018 (0xba52 bytes) 2024-04-23T21:21:29.563 INFO:teuthology.orchestra.run.smithi111.stdout:483 read 0x8ed4c27 thru 0x8edb977 (0x6d51 bytes) 2024-04-23T21:21:29.563 INFO:teuthology.orchestra.run.smithi111.stdout:484 write 0x90e6646 thru 0x90e7e39 (0x17f4 bytes) 2024-04-23T21:21:29.567 INFO:teuthology.orchestra.run.smithi111.stdout:485 punch from 0x3c8c7a to 0x3d1ca8, (0x902e bytes) 2024-04-23T21:21:29.574 INFO:teuthology.orchestra.run.smithi111.stdout:486 writesame 0xb151c6a thru 0xb15a21b data_size 0x6d(0x85b2 bytes) 2024-04-23T21:21:29.574 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.574 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.575 INFO:teuthology.orchestra.run.smithi111.stdout:491 read 0x78db54f thru 0x78e7334 (0xbde6 bytes) 2024-04-23T21:21:29.575 INFO:teuthology.orchestra.run.smithi111.stdout:494 write 0x2e9062a thru 0x2e945e4 (0x3fbb bytes) 2024-04-23T21:21:29.580 INFO:teuthology.orchestra.run.smithi111.stdout:496 read 0x1436cca thru 0x1441054 (0xa38b bytes) 2024-04-23T21:21:29.580 INFO:teuthology.orchestra.run.smithi111.stdout:497 write 0x891307a thru 0x891ae56 (0x7ddd bytes) 2024-04-23T21:21:29.585 INFO:teuthology.orchestra.run.smithi111.stdout:502 read 0x1cf7f53 thru 0x1d05707 (0xd7b5 bytes) 2024-04-23T21:21:29.586 INFO:teuthology.orchestra.run.smithi111.stdout:503 read 0x3f362da thru 0x3f39d38 (0x3a5f bytes) 2024-04-23T21:21:29.586 INFO:teuthology.orchestra.run.smithi111.stdout:505 write 0x3d0e384 thru 0x3d1e233 (0xfeb0 bytes) 2024-04-23T21:21:29.594 INFO:teuthology.orchestra.run.smithi111.stdout:507 writesame 0xa3d75d8 thru 0xa3e2dde data_size 0x1(0xb807 bytes) 2024-04-23T21:21:29.612 INFO:teuthology.orchestra.run.smithi111.stdout:508 write 0x1f3d0a1 thru 0x1f4942a (0xc38a bytes) 2024-04-23T21:21:29.616 INFO:teuthology.orchestra.run.smithi111.stdout:509 read 0x3b5a59e thru 0x3b621c0 (0x7c23 bytes) 2024-04-23T21:21:29.617 INFO:teuthology.orchestra.run.smithi111.stdout:510 read 0xc72c0a thru 0xc7a9aa (0x7da1 bytes) 2024-04-23T21:21:29.617 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.617 INFO:teuthology.orchestra.run.smithi111.stdout:512 read 0x2930695 thru 0x2932120 (0x1a8c bytes) 2024-04-23T21:21:29.617 INFO:teuthology.orchestra.run.smithi111.stdout:513 writesame 0x4167f9b thru 0x416ad5b data_size 0x11(0x2dc1 bytes) 2024-04-23T21:21:29.628 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:29.629+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:29.631 INFO:teuthology.orchestra.run.smithi111.stdout:514 trunc from 0xb15a21c to 0x23c1a85 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7504 3.3 3:c3051774:::rbd_data.111830ac55a1.0000000000000006:head [stat out=16b,truncate 3735552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.640596+0000", 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00271062, 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002727625, 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.642 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7504 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640596+0000", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640596+0000", 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640597+0000", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.850000000000001e-07 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640602+0000", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7740000000000004e-06 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640616+0000", 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4115e-05 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640624+0000", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9750000000000007e-06 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.641121+0000", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000497348 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.641192+0000", 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0672000000000004e-05 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.641355+0000", 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016338299999999999 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.642676+0000", 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001321214 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:21:29.646 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:29.651 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:29.659 INFO:teuthology.orchestra.run.smithi111.stdout:515 write 0x569a5c7 thru 0x569ef52 (0x498c bytes) 2024-04-23T21:21:29.662 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.662 INFO:teuthology.orchestra.run.smithi111.stdout:518 read 0x4499655 thru 0x44a27df (0x918b bytes) 2024-04-23T21:21:29.671 INFO:teuthology.orchestra.run.smithi111.stdout:519 trunc from 0x569ef53 to 0xc9adbba 2024-04-23T21:21:29.676 INFO:teuthology.orchestra.run.smithi111.stdout:520 read 0x5acd6c5 thru 0x5adb11b (0xda57 bytes) 2024-04-23T21:21:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:522 read 0xbfe5dba thru 0xbff0fb0 (0xb1f7 bytes) 2024-04-23T21:21:29.677 INFO:teuthology.orchestra.run.smithi111.stdout:524 writesame 0xb40d6c9 thru 0xb4111f2 data_size 0x1(0x3b2a bytes) 2024-04-23T21:21:29.685 INFO:teuthology.orchestra.run.smithi111.stdout:525 trunc from 0xc9adbba to 0xaf911d0 2024-04-23T21:21:29.701 INFO:teuthology.orchestra.run.smithi111.stdout:526 write 0xdf57dbe thru 0xdf636a4 (0xb8e7 bytes) 2024-04-23T21:21:29.709 INFO:teuthology.orchestra.run.smithi111.stdout:527 writesame 0xbaf0e77 thru 0xbb006ad data_size 0x3b(0xf837 bytes) 2024-04-23T21:21:29.713 INFO:teuthology.orchestra.run.smithi111.stdout:529 read 0xdcda359 thru 0xdce27fa (0x84a2 bytes) 2024-04-23T21:21:29.713 INFO:teuthology.orchestra.run.smithi111.stdout:530 writesame 0x79de088 thru 0x79de9f3 data_size 0x9(0x96c bytes) 2024-04-23T21:21:29.716 INFO:teuthology.orchestra.run.smithi111.stdout:533 read 0xb196b29 thru 0xb1a4ed9 (0xe3b1 bytes) 2024-04-23T21:21:29.717 INFO:teuthology.orchestra.run.smithi111.stdout:534 punch from 0xa2f29af to 0xa2fa6e9, (0x7d3a bytes) 2024-04-23T21:21:29.726 INFO:teuthology.orchestra.run.smithi111.stdout:535 trunc from 0xdf636a5 to 0x910657d 2024-04-23T21:21:29.729 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:29.730+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:29.737 INFO:teuthology.orchestra.run.smithi111.stdout:536 read 0x3b455de thru 0x3b47e7c (0x289f bytes) 2024-04-23T21:21:29.737 INFO:teuthology.orchestra.run.smithi111.stdout:537 read 0x1ce5ce1 thru 0x1cee3da (0x86fa bytes) 2024-04-23T21:21:29.750 INFO:teuthology.orchestra.run.smithi111.stdout:539 write 0xe4851c4 thru 0xe494cff (0xfb3c bytes) 2024-04-23T21:21:29.757 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.766 INFO:teuthology.orchestra.run.smithi111.stdout:542 trunc from 0xe494d00 to 0x6e3aa3b 2024-04-23T21:21:29.788 INFO:teuthology.orchestra.run.smithi111.stdout:544 write 0x9811fa6 thru 0x981d9dd (0xba38 bytes) 2024-04-23T21:21:29.802 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:29.803 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:29.803 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:29.803 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:29.803 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:29.804 INFO:teuthology.orchestra.run.smithi111.stdout:545 trunc from 0x981d9de to 0x1cfbdb8 2024-04-23T21:21:29.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:29.824 INFO:teuthology.orchestra.run.smithi111.stdout:546 read 0xed70a2 thru 0xee421a (0xd179 bytes) 2024-04-23T21:21:29.831 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:29.831+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:29.846 INFO:teuthology.orchestra.run.smithi111.stdout:547 writesame 0xe811cb0 thru 0xe819e1f data_size 0x6d(0x8170 bytes) 2024-04-23T21:21:29.846 INFO:teuthology.orchestra.run.smithi111.stdout:548 write 0x3ef498b thru 0x3f01c5a (0xd2d0 bytes) 2024-04-23T21:21:29.853 INFO:teuthology.orchestra.run.smithi111.stdout:552 write 0x58d977b thru 0x58e06c5 (0x6f4b bytes) 2024-04-23T21:21:29.859 INFO:teuthology.orchestra.run.smithi111.stdout:553 write 0xd0d777a thru 0xd0dad70 (0x35f7 bytes) 2024-04-23T21:21:29.865 INFO:teuthology.orchestra.run.smithi111.stdout:554 read 0xacf4764 thru 0xad02a05 (0xe2a2 bytes) 2024-04-23T21:21:29.866 INFO:teuthology.orchestra.run.smithi111.stdout:555 write 0x953f12 thru 0x961391 (0xd480 bytes) 2024-04-23T21:21:29.869 INFO:teuthology.orchestra.run.smithi111.stdout:558 writesame 0x2b58d64 thru 0x2b655f9 data_size 0x82(0xc896 bytes) 2024-04-23T21:21:29.874 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:29.874 INFO:teuthology.orchestra.run.smithi111.stdout:561 write 0x4378432 thru 0x437b171 (0x2d40 bytes) 2024-04-23T21:21:29.881 INFO:teuthology.orchestra.run.smithi111.stdout:562 writesame 0x56eb442 thru 0x56fb060 data_size 0x2b(0xfc1f bytes) 2024-04-23T21:21:29.886 INFO:teuthology.orchestra.run.smithi111.stdout:563 read 0xc721a73 thru 0xc728c8f (0x721d bytes) 2024-04-23T21:21:29.893 INFO:teuthology.orchestra.run.smithi111.stdout:567 trunc from 0xe819e20 to 0x852aa3d 2024-04-23T21:21:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:569 writesame 0xa1ab83f thru 0xa1b0ceb data_size 0x1(0x54ad bytes) 2024-04-23T21:21:29.914 INFO:teuthology.orchestra.run.smithi111.stdout:570 writesame 0x4de5795 thru 0x4df18da data_size 0x1a(0xc146 bytes) 2024-04-23T21:21:29.920 INFO:teuthology.orchestra.run.smithi111.stdout:571 punch from 0x62b01ff to 0x62b57f5, (0x55f6 bytes) 2024-04-23T21:21:29.920 INFO:teuthology.orchestra.run.smithi111.stdout:572 write 0x6485d89 thru 0x648d6e9 (0x7961 bytes) 2024-04-23T21:21:29.927 INFO:teuthology.orchestra.run.smithi111.stdout:573 trunc from 0xa1b0cec to 0xa1a198f 2024-04-23T21:21:29.927 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:29.927 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:29.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3136 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3080192~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5098836010000003, 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013839865, 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3136 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.928 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:29.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417370+0000", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417377+0000", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6030000000000001e-06 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417381+0000", 2024-04-23T21:21:29.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3030000000000001e-06 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431018+0000", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013636540000000001 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431101+0000", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3014000000000002e-05 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431208+0000", 2024-04-23T21:21:29.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010699199999999999 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3137 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3145728~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5098640919999999, 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014041269, 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3137 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417394+0000", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.55e-06 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417401+0000", 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5930000000000002e-06 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417406+0000", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5059999999999997e-06 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431251+0000", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013845245000000001 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431327+0000", 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6409000000000007e-05 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431429+0000", 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101582 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3138 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3211264~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5098396139999997, 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014240901, 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3138 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8099999999999998e-07 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417419+0000", 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8979999999999999e-06 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417426+0000", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5100000000000004e-06 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417431+0000", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7940000000000002e-06 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431471+0000", 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014040336 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431551+0000", 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0343e-05 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431653+0000", 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010163899999999999 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3139 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3276800~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5098146610000001, 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014372180999999999, 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3139 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417439+0000", 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8950000000000001e-06 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417453+0000", 2024-04-23T21:21:29.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3383e-05 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417458+0000", 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:21:29.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431688+0000", 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014230456000000001 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431755+0000", 2024-04-23T21:21:29.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7183999999999999e-05 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431809+0000", 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3816000000000001e-05 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3140 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3342336~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5097868439999997, 2024-04-23T21:21:29.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014584774999999999, 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3140 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417467+0000", 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:29.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417473+0000", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2559999999999999e-06 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417478+0000", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3510000000000004e-06 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431863+0000", 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014384868 2024-04-23T21:21:29.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431944+0000", 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1433000000000005e-05 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432050+0000", 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105503 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3141 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5097647729999997, 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014710842999999999, 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3141 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417489+0000", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7269999999999999e-06 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417495+0000", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2060000000000004e-06 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417506+0000", 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1277000000000001e-05 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432144+0000", 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014637281 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432198+0000", 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3965999999999998e-05 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3142 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.958 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5097382960000001, 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014717164, 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3142 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417514+0000", 2024-04-23T21:21:29.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417516+0000", 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.875e-06 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417522+0000", 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4169999999999997e-06 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417526+0000", 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4410000000000003e-06 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432223+0000", 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014696300000000001 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432231+0000", 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7409999999999992e-06 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3143 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5097187930000002, 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730144000000001, 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3143 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.963 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:29.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417535+0000", 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7090000000000001e-06 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417541+0000", 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2110000000000003e-06 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417551+0000", 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9250000000000007e-06 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432251+0000", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014700210999999999 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432263+0000", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1715999999999999e-05 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3144 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:29.967 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5096939430000003, 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014732579000000001, 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3144 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.46e-07 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417560+0000", 2024-04-23T21:21:29.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.945e-06 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417566+0000", 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1909999999999997e-06 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417571+0000", 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432283+0000", 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014711864999999999 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432290+0000", 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.525e-06 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3145 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5096745780000003, 2024-04-23T21:21:29.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014738942, 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3145 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417580+0000", 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.125e-06 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417586+0000", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3339999999999998e-06 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417591+0000", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6809999999999997e-06 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432309+0000", 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014718916 2024-04-23T21:21:29.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432316+0000", 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3146 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5096472639999998, 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014741944, 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.975 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3146 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417605+0000", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417607+0000", 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417612+0000", 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5079999999999996e-06 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417617+0000", 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1919999999999998e-06 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432340+0000", 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014723234 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432346+0000", 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4910000000000002e-06 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3147 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5096284119999996, 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014787737, 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3147 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417624+0000", 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417626+0000", 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.012e-06 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417632+0000", 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0220000000000001e-06 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417636+0000", 2024-04-23T21:21:29.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3000000000000003e-06 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432366+0000", 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730115 2024-04-23T21:21:29.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432411+0000", 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4895000000000003e-05 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.985 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3148 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:29.986 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5096097589999999, 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014811695999999999, 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3148 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6100000000000002e-07 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417650+0000", 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1019999999999997e-06 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417658+0000", 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1189999999999997e-06 2024-04-23T21:21:29.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417664+0000", 2024-04-23T21:21:29.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:21:29.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432443+0000", 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014779336000000001 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432454+0000", 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0713999999999999e-05 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3149 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5095808929999999, 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014815238, 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:29.993 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3149 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:29.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3500000000000002e-07 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417673+0000", 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:29.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417679+0000", 2024-04-23T21:21:29.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4530000000000001e-06 2024-04-23T21:21:29.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417683+0000", 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2560000000000004e-06 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432479+0000", 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014796236000000001 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:29.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432486+0000", 2024-04-23T21:21:29.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7479999999999996e-06 2024-04-23T21:21:29.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:29.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:29.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3150 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5095624440000002, 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014825331000000001, 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:30.000 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3150 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417690+0000", 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1600000000000001e-07 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417696+0000", 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3169999999999998e-06 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417703+0000", 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1729999999999998e-06 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417708+0000", 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.481e-06 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432508+0000", 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014799827999999999 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432515+0000", 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160000000000003e-06 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3151 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5095374599999998, 2024-04-23T21:21:30.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014864755, 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3151 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417715+0000", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417717+0000", 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2500000000000001e-06 2024-04-23T21:21:30.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417724+0000", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.289e-06 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417729+0000", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650000000000002e-06 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432567+0000", 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014837845000000001 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432579+0000", 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2334000000000001e-05 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3152 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5095164859999999, 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014901721, 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3152 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417736+0000", 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.96e-07 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417738+0000", 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0540000000000002e-06 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417743+0000", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2510000000000002e-06 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417755+0000", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2536e-05 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432629+0000", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014873569 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432637+0000", 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9149999999999996e-06 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:4908 3.3 3:c0665d5c:::rbd_data.11181344e3fc.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 11010048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.7402477219999999, 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030900347000000002, 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:30.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4908 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187005+0000", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9800000000000002e-07 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198009+0000", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011004491999999999 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198032+0000", 2024-04-23T21:21:30.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2753000000000001e-05 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198059+0000", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6792e-05 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198122+0000", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3633999999999994e-05 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198236+0000", 2024-04-23T21:21:30.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011307 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.199108+0000", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087266500000000001 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.210413+0000", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011304611000000001 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217815+0000", 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0074019869999999996 2024-04-23T21:21:30.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217856+0000", 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1411e-05 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217904+0000", 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8034000000000001e-05 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.017 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4935 3.1b e26/18 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:head v 26'8, mlcod=24'5)", 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.6842956569999998, 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024520146999999999, 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242957+0000", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7800000000000001e-07 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259689+0000", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016732883 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259714+0000", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4992000000000001e-05 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259726+0000", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1916000000000001e-05 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259814+0000", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7964e-05 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:30.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259834+0000", 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0083999999999999e-05 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267434+0000", 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0075992239999999999 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267476+0000", 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2506000000000003e-05 2024-04-23T21:21:30.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4953 3.1b e26/18 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:head v 26'9, mlcod=25'7)", 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.647476014, 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024344686000000001, 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279777+0000", 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.75e-06 2024-04-23T21:21:30.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295820+0000", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016042178000000001 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295845+0000", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5225e-05 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295854+0000", 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.064e-06 2024-04-23T21:21:30.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295951+0000", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6806999999999997e-05 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295969+0000", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8672999999999999e-05 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304074+0000", 2024-04-23T21:21:30.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081043190000000005 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304120+0000", 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6669999999999999e-05 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:30.026 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:30.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:30.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:575 write 0x32dd1d6 thru 0x32e8430 (0xb25b bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:576 writesame 0xeba38cd thru 0xebab591 data_size 0x5b(0x7cc5 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:577 read 0x66821db thru 0x668ea15 (0xc83b bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:578 writesame 0xb08f0fd thru 0xb0917c2 data_size 0x7(0x26c6 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:579 punch from 0x4826366 to 0x48354fe, (0xf198 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:580 write 0x202a164 thru 0x202ed3e (0x4bdb bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:584 read 0xaf60887 thru 0xaf703cb (0xfb45 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:585 read 0xd5621cd thru 0xd56f592 (0xd3c6 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:586 writesame 0x2030990 thru 0x203ae8c data_size 0x13(0xa4fd bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:588 writesame 0x4d2e8cf thru 0x4d34edd data_size 0x3(0x660f bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:589 write 0x8ffe065 thru 0x9001726 (0x36c2 bytes) 2024-04-23T21:21:30.031 INFO:teuthology.orchestra.run.smithi111.stdout:592 punch from 0x338dc3c to 0x339a764, (0xcb28 bytes) 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:593 read 0xcb3673e thru 0xcb45e2b (0xf6ee bytes) 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:594 punch from 0x80f1529 to 0x80fcd68, (0xb83f bytes) 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:595 write 0x3a1d87e thru 0x3a20f15 (0x3698 bytes) 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:596 punch from 0xdb6f41 to 0xdc4644, (0xd703 bytes) 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:597 trunc from 0xebab592 to 0x160dc86 2024-04-23T21:21:30.032 INFO:teuthology.orchestra.run.smithi111.stdout:598 write 0xdfe95aa thru 0xdff69db (0xd432 bytes) 2024-04-23T21:21:30.032 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:30.028+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:30.039 INFO:teuthology.orchestra.run.smithi111.stdout:600 trunc from 0xdff69dc to 0x5f8d5a9 2024-04-23T21:21:30.055 INFO:teuthology.orchestra.run.smithi111.stdout:601 trunc from 0x5f8d5a9 to 0xbe2ce15 2024-04-23T21:21:30.060 INFO:teuthology.orchestra.run.smithi111.stdout:602 punch from 0x92f3550 to 0x92f3a6c, (0x51c bytes) 2024-04-23T21:21:30.060 INFO:teuthology.orchestra.run.smithi111.stdout:603 write 0x91b0b16 thru 0x91b1fa6 (0x1491 bytes) 2024-04-23T21:21:30.063 INFO:teuthology.orchestra.run.smithi111.stdout:605 write 0x487d368 thru 0x4886cb9 (0x9952 bytes) 2024-04-23T21:21:30.069 INFO:teuthology.orchestra.run.smithi111.stdout:606 write 0x7f746ec thru 0x7f7967b (0x4f90 bytes) 2024-04-23T21:21:30.076 INFO:teuthology.orchestra.run.smithi111.stdout:607 punch from 0x66a7cd3 to 0x66afdd0, (0x80fd bytes) 2024-04-23T21:21:30.080 INFO:teuthology.orchestra.run.smithi111.stdout:609 trunc from 0xbe2ce15 to 0x80bb2b4 2024-04-23T21:21:30.088 INFO:teuthology.orchestra.run.smithi111.stdout:610 read 0x2060341 thru 0x206a7ec (0xa4ac bytes) 2024-04-23T21:21:30.089 INFO:teuthology.orchestra.run.smithi111.stdout:611 write 0x3189fcb thru 0x318f9ea (0x5a20 bytes) 2024-04-23T21:21:30.092 INFO:teuthology.orchestra.run.smithi111.stdout:613 punch from 0x5088c07 to 0x5095cea, (0xd0e3 bytes) 2024-04-23T21:21:30.092 INFO:teuthology.orchestra.run.smithi111.stdout:614 writesame 0xdac82b thru 0xdba64b data_size 0x55(0xde21 bytes) 2024-04-23T21:21:30.104 INFO:teuthology.orchestra.run.smithi111.stdout:615 write 0xb2db233 thru 0xb2e6f81 (0xbd4f bytes) 2024-04-23T21:21:30.108 INFO:teuthology.orchestra.run.smithi111.stdout:616 read 0x3dbbe0d thru 0x3dc2764 (0x6958 bytes) 2024-04-23T21:21:30.118 INFO:teuthology.orchestra.run.smithi111.stdout:617 write 0xd8e7a4a thru 0xd8f14c7 (0x9a7e bytes) 2024-04-23T21:21:30.126 INFO:teuthology.orchestra.run.smithi111.stdout:618 writesame 0x1ca8 thru 0x20af data_size 0x4(0x408 bytes) 2024-04-23T21:21:30.127 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.127 INFO:teuthology.orchestra.run.smithi111.stdout:620 write 0x7ad1c62 thru 0x7adc9d5 (0xad74 bytes) 2024-04-23T21:21:30.128 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:30.129+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:30.136 INFO:teuthology.orchestra.run.smithi111.stdout:622 trunc from 0xd8f14c8 to 0x904d5b7 2024-04-23T21:21:30.146 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:30.146 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:30.146 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:30.146 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:30.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:30.159 INFO:teuthology.orchestra.run.smithi111.stdout:623 write 0xbfe68bc thru 0xbff544b (0xeb90 bytes) 2024-04-23T21:21:30.165 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.165 INFO:teuthology.orchestra.run.smithi111.stdout:625 write 0x3a1e58 thru 0x3a4f76 (0x311f bytes) 2024-04-23T21:21:30.165 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:30.166 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:30.166 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:30.166 INFO:teuthology.orchestra.run.smithi111.stdout:626 punch from 0x3a48160 to 0x3a49fee, (0x1e8e bytes) 2024-04-23T21:21:30.166 INFO:teuthology.orchestra.run.smithi111.stdout:627 write 0x5c95ccd thru 0x5ca51f8 (0xf52c bytes) 2024-04-23T21:21:30.173 INFO:teuthology.orchestra.run.smithi111.stdout:629 read 0x4e3e5d5 thru 0x4e4b07f (0xcaab bytes) 2024-04-23T21:21:30.174 INFO:teuthology.orchestra.run.smithi111.stdout:631 punch from 0x1897364 to 0x189f3de, (0x807a bytes) 2024-04-23T21:21:30.174 INFO:teuthology.orchestra.run.smithi111.stdout:632 writesame 0x71ab6e2 thru 0x71aeee5 data_size 0x1e(0x3804 bytes) 2024-04-23T21:21:30.178 INFO:teuthology.orchestra.run.smithi111.stdout:633 trunc from 0xbff544c to 0xd054f79 2024-04-23T21:21:30.183 INFO:teuthology.orchestra.run.smithi111.stdout:634 read 0x8a6a399 thru 0x8a7849e (0xe106 bytes) 2024-04-23T21:21:30.184 INFO:teuthology.orchestra.run.smithi111.stdout:635 punch from 0x4668574 to 0x4670e24, (0x88b0 bytes) 2024-04-23T21:21:30.184 INFO:teuthology.orchestra.run.smithi111.stdout:636 read 0xa64ad25 thru 0xa64bca8 (0xf84 bytes) 2024-04-23T21:21:30.184 INFO:teuthology.orchestra.run.smithi111.stdout:637 read 0x2f31fbb thru 0x2f3dd53 (0xbd99 bytes) 2024-04-23T21:21:30.185 INFO:teuthology.orchestra.run.smithi111.stdout:639 trunc from 0xd054f79 to 0xcb74022 2024-04-23T21:21:30.186 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:30.186 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:30.191 INFO:teuthology.orchestra.run.smithi111.stdout:641 read 0x2e39bb0 thru 0x2e47425 (0xd876 bytes) 2024-04-23T21:21:30.192 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.192 INFO:teuthology.orchestra.run.smithi111.stdout:646 punch from 0x64d7e25 to 0x64e1953, (0x9b2e bytes) 2024-04-23T21:21:30.192 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.192 INFO:teuthology.orchestra.run.smithi111.stdout:650 writesame 0x22140bd thru 0x221af95 data_size 0x9(0x6ed9 bytes) 2024-04-23T21:21:30.194 INFO:teuthology.orchestra.run.smithi111.stdout:651 read 0x508c747 thru 0x5097615 (0xaecf bytes) 2024-04-23T21:21:30.194 INFO:teuthology.orchestra.run.smithi111.stdout:652 write 0x5159f36 thru 0x5168d7f (0xee4a bytes) 2024-04-23T21:21:30.199 INFO:teuthology.orchestra.run.smithi111.stdout:655 read 0x2d85dda thru 0x2d8e20a (0x8431 bytes) 2024-04-23T21:21:30.199 INFO:teuthology.orchestra.run.smithi111.stdout:656 write 0x4c82769 thru 0x4c86a0a (0x42a2 bytes) 2024-04-23T21:21:30.203 INFO:teuthology.orchestra.run.smithi111.stdout:658 write 0x906c259 thru 0x90709b7 (0x475f bytes) 2024-04-23T21:21:30.204 INFO:teuthology.orchestra.run.smithi111.stdout:660 punch from 0x49a48ca to 0x49a4d51, (0x487 bytes) 2024-04-23T21:21:30.204 INFO:teuthology.orchestra.run.smithi111.stdout:661 trunc from 0xcb74022 to 0xccc4422 2024-04-23T21:21:30.205 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:30.205 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:30.212 INFO:teuthology.orchestra.run.smithi111.stdout:662 trunc from 0xccc4422 to 0xb1a51b2 2024-04-23T21:21:30.219 INFO:teuthology.orchestra.run.smithi111.stdout:664 read 0x495dc6 thru 0x4966e9 (0x924 bytes) 2024-04-23T21:21:30.225 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:30.225 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:30.229 INFO:teuthology.orchestra.run.smithi111.stdout:665 write 0xdcd08ce thru 0xdcdb71f (0xae52 bytes) 2024-04-23T21:21:30.233 INFO:teuthology.orchestra.run.smithi111.stdout:666 trunc from 0xdcdb720 to 0xdf873e7 2024-04-23T21:21:30.238 INFO:teuthology.orchestra.run.smithi111.stdout:667 writesame 0x443361d thru 0x4442879 data_size 0x1(0xf25d bytes) 2024-04-23T21:21:30.239 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:30.240+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:30.244 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:30.244 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:30.255 INFO:teuthology.orchestra.run.smithi111.stdout:668 read 0x2ead822 thru 0x2ebd083 (0xf862 bytes) 2024-04-23T21:21:30.256 INFO:teuthology.orchestra.run.smithi111.stdout:670 write 0x45cbe85 thru 0x45d0293 (0x440f bytes) 2024-04-23T21:21:30.257 INFO:teuthology.orchestra.run.smithi111.stdout:672 write 0x96d6949 thru 0x96d97b8 (0x2e70 bytes) 2024-04-23T21:21:30.261 INFO:teuthology.orchestra.run.smithi111.stdout:673 read 0x7b35649 thru 0x7b38c77 (0x362f bytes) 2024-04-23T21:21:30.261 INFO:teuthology.orchestra.run.smithi111.stdout:674 punch from 0x977d73f to 0x977f392, (0x1c53 bytes) 2024-04-23T21:21:30.261 INFO:teuthology.orchestra.run.smithi111.stdout:675 write 0x32d66d2 thru 0x32e4e94 (0xe7c3 bytes) 2024-04-23T21:21:30.264 INFO:teuthology.orchestra.run.smithi111.stdout:676 write 0x18bf708 thru 0x18c33ca (0x3cc3 bytes) 2024-04-23T21:21:30.269 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:30.270 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:30.276 INFO:teuthology.orchestra.run.smithi111.stdout:677 trunc from 0xdf873e7 to 0x553c2d6 2024-04-23T21:21:30.290 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:30.291 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:30.294 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:30.303 INFO:teuthology.orchestra.run.smithi111.stdout:678 writesame 0xa85939a thru 0xa85c5c7 data_size 0x3(0x322e bytes) 2024-04-23T21:21:30.303 INFO:teuthology.orchestra.run.smithi111.stdout:680 read 0x62313a6 thru 0x6238f0f (0x7b6a bytes) 2024-04-23T21:21:30.303 INFO:teuthology.orchestra.run.smithi111.stdout:682 writesame 0x7946578 thru 0x7950819 data_size 0x51(0xa2a2 bytes) 2024-04-23T21:21:30.308 INFO:teuthology.orchestra.run.smithi111.stdout:683 write 0xa13f3ee thru 0xa14ad82 (0xb995 bytes) 2024-04-23T21:21:30.311 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:30.311 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:30.312 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.320 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:30.321 INFO:teuthology.orchestra.run.smithi111.stdout:686 write 0xd6c13fe thru 0xd6c20e3 (0xce6 bytes) 2024-04-23T21:21:30.324 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.324 INFO:teuthology.orchestra.run.smithi111.stdout:688 read 0xbed151 thru 0xbf6a2c (0x98dc bytes) 2024-04-23T21:21:30.325 INFO:teuthology.orchestra.run.smithi111.stdout:689 punch from 0x9153b4 to 0x91b4b9, (0x6105 bytes) 2024-04-23T21:21:30.325 INFO:teuthology.orchestra.run.smithi111.stdout:690 punch from 0x9ab4e9a to 0x9ac263f, (0xd7a5 bytes) 2024-04-23T21:21:30.325 INFO:teuthology.orchestra.run.smithi111.stdout:691 read 0x87bc0e6 thru 0x87bc234 (0x14f bytes) 2024-04-23T21:21:30.325 INFO:teuthology.orchestra.run.smithi111.stdout:692 read 0xd1ac948 thru 0xd1afe79 (0x3532 bytes) 2024-04-23T21:21:30.325 INFO:teuthology.orchestra.run.smithi111.stdout:693 read 0x84c20d5 thru 0x84c55e0 (0x350c bytes) 2024-04-23T21:21:30.326 INFO:teuthology.orchestra.run.smithi111.stdout:694 writesame 0x883fe0e thru 0x8844791 data_size 0xa(0x4984 bytes) 2024-04-23T21:21:30.329 INFO:teuthology.orchestra.run.smithi111.stdout:695 punch from 0x2c7373 to 0x2cba83, (0x4710 bytes) 2024-04-23T21:21:30.329 INFO:teuthology.orchestra.run.smithi111.stdout:696 read 0xba54e3f thru 0xba63557 (0xe719 bytes) 2024-04-23T21:21:30.329 INFO:teuthology.orchestra.run.smithi111.stdout:698 writesame 0x8644f63 thru 0x8645801 data_size 0x1(0x89f bytes) 2024-04-23T21:21:30.330 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:30.330+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:30.331 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.331 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.332 INFO:teuthology.orchestra.run.smithi111.stdout:701 read 0x7ba3790 thru 0x7bafaa5 (0xc316 bytes) 2024-04-23T21:21:30.332 INFO:teuthology.orchestra.run.smithi111.stdout:703 punch from 0x1b233d3 to 0x1b2d3e0, (0xa00d bytes) 2024-04-23T21:21:30.333 INFO:teuthology.orchestra.run.smithi111.stdout:704 read 0xc51d799 thru 0xc5209a2 (0x320a bytes) 2024-04-23T21:21:30.333 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.333 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.333 INFO:teuthology.orchestra.run.smithi111.stdout:711 write 0x9127862 thru 0x912e7ed (0x6f8c bytes) 2024-04-23T21:21:30.338 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.338 INFO:teuthology.orchestra.run.smithi111.stdout:714 punch from 0x563925f to 0x563f4aa, (0x624b bytes) 2024-04-23T21:21:30.338 INFO:teuthology.orchestra.run.smithi111.stdout:715 write 0x87566e2 thru 0x875dd63 (0x7682 bytes) 2024-04-23T21:21:30.339 INFO:teuthology.orchestra.run.smithi111.stdout:716 trunc from 0xd6c20e4 to 0xb95d854 2024-04-23T21:21:30.351 INFO:teuthology.orchestra.run.smithi111.stdout:718 flatten 2024-04-23T21:21:30.359 INFO:teuthology.orchestra.run.smithi111.stdout:719 write 0x1c87955 thru 0x1c93c75 (0xc321 bytes) 2024-04-23T21:21:30.370 INFO:teuthology.orchestra.run.smithi111.stdout:721 write 0xbd74f87 thru 0xbd79f62 (0x4fdc bytes) 2024-04-23T21:21:30.373 INFO:teuthology.orchestra.run.smithi111.stdout:722 read 0x9aa7d14 thru 0x9aaa749 (0x2a36 bytes) 2024-04-23T21:21:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:728 read 0x88a5ab6 thru 0x88b4782 (0xeccd bytes) 2024-04-23T21:21:30.374 INFO:teuthology.orchestra.run.smithi111.stdout:729 writesame 0xba80c17 thru 0xba8a554 data_size 0x5(0x993e bytes) 2024-04-23T21:21:30.379 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.387 INFO:teuthology.orchestra.run.smithi111.stdout:731 write 0xd406715 thru 0xd40f68b (0x8f77 bytes) 2024-04-23T21:21:30.392 INFO:teuthology.orchestra.run.smithi111.stdout:732 write 0x2442cfe thru 0x2452ba7 (0xfeaa bytes) 2024-04-23T21:21:30.396 INFO:teuthology.orchestra.run.smithi111.stdout:733 write 0xd30c5d thru 0xd3382b (0x2bcf bytes) 2024-04-23T21:21:30.398 INFO:teuthology.orchestra.run.smithi111.stdout:734 read 0x7379cdc thru 0x7388464 (0xe789 bytes) 2024-04-23T21:21:30.398 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.398 INFO:teuthology.orchestra.run.smithi111.stdout:737 write 0xf4bdf3 thru 0xf5ab7e (0xed8c bytes) 2024-04-23T21:21:30.402 INFO:teuthology.orchestra.run.smithi111.stdout:739 writesame 0xc3f1396 thru 0xc3fdbda data_size 0x1(0xc845 bytes) 2024-04-23T21:21:30.431 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:30.431+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:30.434 INFO:teuthology.orchestra.run.smithi111.stdout:740 trunc from 0xd40f68c to 0x25dca 2024-04-23T21:21:30.476 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:30.476 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:30.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:30.488 INFO:teuthology.orchestra.run.smithi111.stdout:742 write 0x6d2b07f thru 0x6d3749d (0xc41f bytes) 2024-04-23T21:21:30.492 INFO:teuthology.orchestra.run.smithi111.stdout:743 read 0x4dc545d thru 0x4dc5b62 (0x706 bytes) 2024-04-23T21:21:30.492 INFO:teuthology.orchestra.run.smithi111.stdout:745 read 0x1605cba thru 0x1610939 (0xac80 bytes) 2024-04-23T21:21:30.498 INFO:teuthology.orchestra.run.smithi111.stdout:746 write 0x7981902 thru 0x798576a (0x3e69 bytes) 2024-04-23T21:21:30.501 INFO:teuthology.orchestra.run.smithi111.stdout:747 punch from 0x3d8bd39 to 0x3d93232, (0x74f9 bytes) 2024-04-23T21:21:30.501 INFO:teuthology.orchestra.run.smithi111.stdout:748 write 0x7800d75 thru 0x78034bb (0x2747 bytes) 2024-04-23T21:21:30.504 INFO:teuthology.orchestra.run.smithi111.stdout:749 write 0x2c90d90 thru 0x2c95ebd (0x512e bytes) 2024-04-23T21:21:30.519 INFO:teuthology.orchestra.run.smithi111.stdout:750 write 0xc67cdfe thru 0xc6853e8 (0x85eb bytes) 2024-04-23T21:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout:751 read 0x86ba238 thru 0x86bb011 (0xdda bytes) 2024-04-23T21:21:30.524 INFO:teuthology.orchestra.run.smithi111.stdout:753 write 0xb049b53 thru 0xb04ddda (0x4288 bytes) 2024-04-23T21:21:30.525 INFO:teuthology.orchestra.run.smithi111.stdout:754 read 0x6b55061 thru 0x6b62db1 (0xdd51 bytes) 2024-04-23T21:21:30.526 INFO:teuthology.orchestra.run.smithi111.stdout:755 punch from 0x64b3393 to 0x64bc7e1, (0x944e bytes) 2024-04-23T21:21:30.526 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.526 INFO:teuthology.orchestra.run.smithi111.stdout:761 writesame 0x6d9d85d thru 0x6da1b69 data_size 0x1(0x430d bytes) 2024-04-23T21:21:30.534 INFO:teuthology.orchestra.run.smithi111.stdout:762 write 0x239d2a9 thru 0x239d97b (0x6d3 bytes) 2024-04-23T21:21:30.537 INFO:teuthology.orchestra.run.smithi111.stdout:763 read 0xa85d05e thru 0xa862d78 (0x5d1b bytes) 2024-04-23T21:21:30.537 INFO:teuthology.orchestra.run.smithi111.stdout:764 writesame 0x84e5897 thru 0x84f28ec data_size 0x9(0xd056 bytes) 2024-04-23T21:21:30.542 INFO:teuthology.orchestra.run.smithi111.stdout:765 writesame 0x5f6467d thru 0x5f6dbdd data_size 0x15(0x9561 bytes) 2024-04-23T21:21:30.545 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.545 INFO:teuthology.orchestra.run.smithi111.stdout:767 write 0x9f599e0 thru 0x9f662bb (0xc8dc bytes) 2024-04-23T21:21:30.548 INFO:teuthology.orchestra.run.smithi111.stdout:769 write 0x3886125 thru 0x3892615 (0xc4f1 bytes) 2024-04-23T21:21:30.553 INFO:teuthology.orchestra.run.smithi111.stdout:771 write 0x137db49 thru 0x138aeff (0xd3b7 bytes) 2024-04-23T21:21:30.556 INFO:teuthology.orchestra.run.smithi111.stdout:772 writesame 0x49957d6 thru 0x49a362a data_size 0x2f(0xde55 bytes) 2024-04-23T21:21:30.560 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.560 INFO:teuthology.orchestra.run.smithi111.stdout:775 write 0x9b2a567 thru 0x9b30668 (0x6102 bytes) 2024-04-23T21:21:30.563 INFO:teuthology.orchestra.run.smithi111.stdout:776 punch from 0x782a7c0 to 0x78331a2, (0x89e2 bytes) 2024-04-23T21:21:30.566 INFO:teuthology.orchestra.run.smithi111.stdout:778 trunc from 0xc6853e9 to 0x9826e97 2024-04-23T21:21:30.569 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:30.570+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:30.579 INFO:teuthology.orchestra.run.smithi111.stdout:779 read 0x72ba265 thru 0x72c2830 (0x85cc bytes) 2024-04-23T21:21:30.580 INFO:teuthology.orchestra.run.smithi111.stdout:782 read 0x811f887 thru 0x812dbb3 (0xe32d bytes) 2024-04-23T21:21:30.588 INFO:teuthology.orchestra.run.smithi111.stdout:783 trunc from 0x9826e97 to 0x300cf2d 2024-04-23T21:21:30.590 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:30.590 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:21:30.590 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T21:21:30.590 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.046825 2024-04-23T21:21:30.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 0.0468246037264507 2024-04-23T21:21:30.602 INFO:teuthology.orchestra.run.smithi111.stdout:785 punch from 0x13757e1 to 0x137eb32, (0x9351 bytes) 2024-04-23T21:21:30.602 INFO:teuthology.orchestra.run.smithi111.stdout:787 read 0x3e4c91 thru 0x3ef67d (0xa9ed bytes) 2024-04-23T21:21:30.603 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.619 INFO:teuthology.orchestra.run.smithi111.stdout:789 writesame 0xbe9feb3 thru 0xbea5bb8 data_size 0xe(0x5d06 bytes) 2024-04-23T21:21:30.619 INFO:teuthology.orchestra.run.smithi111.stdout:790 read 0x4a24acf thru 0x4a30900 (0xbe32 bytes) 2024-04-23T21:21:30.620 INFO:teuthology.orchestra.run.smithi111.stdout:792 punch from 0x9aceb55 to 0x9ad52bb, (0x6766 bytes) 2024-04-23T21:21:30.620 INFO:teuthology.orchestra.run.smithi111.stdout:794 punch from 0x5099ddf to 0x509ff70, (0x6191 bytes) 2024-04-23T21:21:30.620 INFO:teuthology.orchestra.run.smithi111.stdout:795 punch from 0xb540895 to 0xb5443ac, (0x3b17 bytes) 2024-04-23T21:21:30.620 INFO:teuthology.orchestra.run.smithi111.stdout:796 read 0x9534748 thru 0x9537a4d (0x3306 bytes) 2024-04-23T21:21:30.621 INFO:teuthology.orchestra.run.smithi111.stdout:798 write 0x68f367 thru 0x697675 (0x830f bytes) 2024-04-23T21:21:30.621 INFO:teuthology.orchestra.run.smithi111.stdout:799 writesame 0x1242635 thru 0x124a38b data_size 0x1(0x7d57 bytes) 2024-04-23T21:21:30.632 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:30.633+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:30.634 INFO:teuthology.orchestra.run.smithi111.stdout:800 read 0x196a587 thru 0x19781ed (0xdc67 bytes) 2024-04-23T21:21:30.634 INFO:teuthology.orchestra.run.smithi111.stdout:801 read 0x6735e73 thru 0x674240e (0xc59c bytes) 2024-04-23T21:21:30.642 INFO:teuthology.orchestra.run.smithi111.stdout:802 write 0xd439630 thru 0xd44308b (0x9a5c bytes) 2024-04-23T21:21:30.643 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:30.643 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:30.647 INFO:teuthology.orchestra.run.smithi111.stdout:804 trunc from 0xd44308c to 0xb89dff9 2024-04-23T21:21:30.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:30.657 INFO:teuthology.orchestra.run.smithi111.stdout:806 read 0x74cbbb0 thru 0x74cf794 (0x3be5 bytes) 2024-04-23T21:21:30.658 INFO:teuthology.orchestra.run.smithi111.stdout:807 writesame 0xb138c3f thru 0xb144421 data_size 0x23(0xb7e3 bytes) 2024-04-23T21:21:30.664 INFO:teuthology.orchestra.run.smithi111.stdout:808 writesame 0x1e925dc thru 0x1e9ad1f data_size 0x16(0x8744 bytes) 2024-04-23T21:21:30.670 INFO:teuthology.orchestra.run.smithi111.stdout:809 write 0x738fb34 thru 0x739b64a (0xbb17 bytes) 2024-04-23T21:21:30.678 INFO:teuthology.orchestra.run.smithi111.stdout:810 write 0x34d3ceb thru 0x34e1ce1 (0xdff7 bytes) 2024-04-23T21:21:30.693 INFO:teuthology.orchestra.run.smithi111.stdout:811 write 0xd85ed43 thru 0xd86484b (0x5b09 bytes) 2024-04-23T21:21:30.700 INFO:teuthology.orchestra.run.smithi111.stdout:814 write 0x1ac5fcc thru 0x1ad0bd7 (0xac0c bytes) 2024-04-23T21:21:30.706 INFO:teuthology.orchestra.run.smithi111.stdout:815 trunc from 0xd86484c to 0xcbab134 2024-04-23T21:21:30.714 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.714 INFO:teuthology.orchestra.run.smithi111.stdout:818 read 0x169ad3c thru 0x16a1325 (0x65ea bytes) 2024-04-23T21:21:30.715 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.715 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:30.715 INFO:teuthology.orchestra.run.smithi111.stdout:822 writesame 0x70f2fe5 thru 0x7100858 data_size 0xe(0xd874 bytes) 2024-04-23T21:21:30.724 INFO:teuthology.orchestra.run.smithi111.stdout:824 write 0xbd1c0df thru 0xbd2acb3 (0xebd5 bytes) 2024-04-23T21:21:30.727 INFO:teuthology.orchestra.run.smithi111.stdout:825 read 0xa1c36d9 thru 0xa1c4d93 (0x16bb bytes) 2024-04-23T21:21:30.727 INFO:teuthology.orchestra.run.smithi111.stdout:826 clone 7 order 19 su 32768 sc 13 2024-04-23T21:21:30.738 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:30.739+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:30.799 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:30.799 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:30.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:30.834 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:30.835+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:30.836 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:21:30.935 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:30.936+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:30.953 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:30.962 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:31.020 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-04-23T21:21:31.022 INFO:teuthology.orchestra.run.smithi111.stdout:827 writesame 0x6f964fc thru 0x6fa51da data_size 0x57(0xecdf bytes) 2024-04-23T21:21:31.033 INFO:teuthology.orchestra.run.smithi111.stdout:828 writesame 0x6c72726 thru 0x6c7aac2 data_size 0xb(0x839d bytes) 2024-04-23T21:21:31.040 INFO:teuthology.orchestra.run.smithi111.stdout:830 write 0x97795db thru 0x9786d6a (0xd790 bytes) 2024-04-23T21:21:31.046 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:31.047+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:31.047 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.053 INFO:teuthology.orchestra.run.smithi111.stdout:833 write 0xdf758a1 thru 0xdf857c5 (0xff25 bytes) 2024-04-23T21:21:31.058 INFO:teuthology.orchestra.run.smithi111.stdout:834 write 0x45d4dba thru 0x45e15e7 (0xc82e bytes) 2024-04-23T21:21:31.064 INFO:teuthology.orchestra.run.smithi111.stdout:835 writesame 0x5a3403a thru 0x5a37269 data_size 0x10(0x3230 bytes) 2024-04-23T21:21:31.069 INFO:teuthology.orchestra.run.smithi111.stdout:836 write 0x3595290 thru 0x35a2aa1 (0xd812 bytes) 2024-04-23T21:21:31.075 INFO:teuthology.orchestra.run.smithi111.stdout:838 write 0x4636a8e thru 0x464565d (0xebd0 bytes) 2024-04-23T21:21:31.079 INFO:teuthology.orchestra.run.smithi111.stdout:839 punch from 0x85d4a98 to 0x85e0cc5, (0xc22d bytes) 2024-04-23T21:21:31.079 INFO:teuthology.orchestra.run.smithi111.stdout:840 write 0x89f4c25 thru 0x89fed25 (0xa101 bytes) 2024-04-23T21:21:31.083 INFO:teuthology.orchestra.run.smithi111.stdout:842 punch from 0x770f0fc to 0x771efe3, (0xfee7 bytes) 2024-04-23T21:21:31.083 INFO:teuthology.orchestra.run.smithi111.stdout:844 read 0x39b6e04 thru 0x39c3cd2 (0xcecf bytes) 2024-04-23T21:21:31.098 INFO:teuthology.orchestra.run.smithi111.stdout:846 trunc from 0xdf857c6 to 0x1ecd1c0 2024-04-23T21:21:31.119 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.119 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:31.147 INFO:teuthology.orchestra.run.smithi111.stdout:848 write 0x8b6e28c thru 0x8b75e04 (0x7b79 bytes) 2024-04-23T21:21:31.151 INFO:teuthology.orchestra.run.smithi111.stdout:850 punch from 0x7cf3470 to 0x7cfdd6c, (0xa8fc bytes) 2024-04-23T21:21:31.151 INFO:teuthology.orchestra.run.smithi111.stdout:851 punch from 0x5490039 to 0x54949e6, (0x49ad bytes) 2024-04-23T21:21:31.151 INFO:teuthology.orchestra.run.smithi111.stdout:852 write 0x223d46 thru 0x22dc33 (0x9eee bytes) 2024-04-23T21:21:31.158 INFO:teuthology.orchestra.run.smithi111.stdout:853 read 0x5d6dc26 thru 0x5d6f055 (0x1430 bytes) 2024-04-23T21:21:31.163 INFO:teuthology.orchestra.run.smithi111.stdout:854 writesame 0x93d566d thru 0x93dda76 data_size 0x1(0x840a bytes) 2024-04-23T21:21:31.163 INFO:teuthology.orchestra.run.smithi111.stdout:855 write 0x2152839 thru 0x215b8a6 (0x906e bytes) 2024-04-23T21:21:31.166 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.166 INFO:teuthology.orchestra.run.smithi111.stdout:857 trunc from 0x93dda77 to 0x92390b2 2024-04-23T21:21:31.171 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:31.172+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:31.171 INFO:teuthology.orchestra.run.smithi111.stdout:859 punch from 0x4ab5704 to 0x4ac0cea, (0xb5e6 bytes) 2024-04-23T21:21:31.176 INFO:teuthology.orchestra.run.smithi111.stdout:860 trunc from 0x92390b2 to 0x49aa99d 2024-04-23T21:21:31.198 INFO:teuthology.orchestra.run.smithi111.stdout:861 trunc from 0x49aa99d to 0xec47836 2024-04-23T21:21:31.204 INFO:teuthology.orchestra.run.smithi111.stdout:862 read 0x81bed7f thru 0x81ccfc5 (0xe247 bytes) 2024-04-23T21:21:31.205 INFO:teuthology.orchestra.run.smithi111.stdout:863 punch from 0xa2f6585 to 0xa2fd427, (0x6ea2 bytes) 2024-04-23T21:21:31.205 INFO:teuthology.orchestra.run.smithi111.stdout:864 write 0xb46a584 thru 0xb46cdf5 (0x2872 bytes) 2024-04-23T21:21:31.207 INFO:teuthology.orchestra.run.smithi111.stdout:867 writesame 0x9ceff62 thru 0x9cffb0c data_size 0x1(0xfbab bytes) 2024-04-23T21:21:31.225 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.225 INFO:teuthology.orchestra.run.smithi111.stdout:871 punch from 0xa5c882 to 0xa6a09e, (0xd81c bytes) 2024-04-23T21:21:31.225 INFO:teuthology.orchestra.run.smithi111.stdout:872 punch from 0x2cf20ca to 0x2cf9c53, (0x7b89 bytes) 2024-04-23T21:21:31.225 INFO:teuthology.orchestra.run.smithi111.stdout:873 read 0x7f5ad79 thru 0x7f6a2f4 (0xf57c bytes) 2024-04-23T21:21:31.238 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:31.238+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:31.239 INFO:teuthology.orchestra.run.smithi111.stdout:874 trunc from 0xec47836 to 0x2bbe304 2024-04-23T21:21:31.250 INFO:teuthology.orchestra.run.smithi111.stdout:875 read 0xef7915 thru 0xf014ce (0x9bba bytes) 2024-04-23T21:21:31.250 INFO:teuthology.orchestra.run.smithi111.stdout:878 punch from 0x173954e to 0x1743568, (0xa01a bytes) 2024-04-23T21:21:31.262 INFO:teuthology.orchestra.run.smithi111.stdout:879 write 0x7886b62 thru 0x78892f3 (0x2792 bytes) 2024-04-23T21:21:31.269 INFO:teuthology.orchestra.run.smithi111.stdout:880 writesame 0x749a52a thru 0x74a56ba data_size 0x1(0xb191 bytes) 2024-04-23T21:21:31.280 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.281 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.281 INFO:teuthology.orchestra.run.smithi111.stdout:881 writesame 0x335dc17 thru 0x33620ed data_size 0x1(0x44d7 bytes) 2024-04-23T21:21:31.288 INFO:teuthology.orchestra.run.smithi111.stdout:882 write 0x6542831 thru 0x654d587 (0xad57 bytes) 2024-04-23T21:21:31.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:31.309 INFO:teuthology.orchestra.run.smithi111.stdout:883 writesame 0xc999eb8 thru 0xc99fd9d data_size 0x3(0x5ee6 bytes) 2024-04-23T21:21:31.309 INFO:teuthology.orchestra.run.smithi111.stdout:884 punch from 0x3f04c13 to 0x3f058f8, (0xce5 bytes) 2024-04-23T21:21:31.310 INFO:teuthology.orchestra.run.smithi111.stdout:885 write 0x7417b1 thru 0x74592e (0x417e bytes) 2024-04-23T21:21:31.314 INFO:teuthology.orchestra.run.smithi111.stdout:886 punch from 0x3bb82f0 to 0x3bbf9ea, (0x76fa bytes) 2024-04-23T21:21:31.314 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.322 INFO:teuthology.orchestra.run.smithi111.stdout:888 write 0xe0cb70e thru 0xe0d406a (0x895d bytes) 2024-04-23T21:21:31.326 INFO:teuthology.orchestra.run.smithi111.stdout:889 read 0x2fd04b9 thru 0x2fd890d (0x8455 bytes) 2024-04-23T21:21:31.327 INFO:teuthology.orchestra.run.smithi111.stdout:891 read 0xc2778fd thru 0xc280263 (0x8967 bytes) 2024-04-23T21:21:31.327 INFO:teuthology.orchestra.run.smithi111.stdout:892 writesame 0x99f42c thru 0x9a5909 data_size 0x1(0x64de bytes) 2024-04-23T21:21:31.339 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:31.340+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:31.342 INFO:teuthology.orchestra.run.smithi111.stdout:893 punch from 0x7d841b7 to 0x7d869ec, (0x2835 bytes) 2024-04-23T21:21:31.342 INFO:teuthology.orchestra.run.smithi111.stdout:894 punch from 0x7170e69 to 0x71794b3, (0x864a bytes) 2024-04-23T21:21:31.342 INFO:teuthology.orchestra.run.smithi111.stdout:895 write 0xc35b0df thru 0xc369192 (0xe0b4 bytes) 2024-04-23T21:21:31.350 INFO:teuthology.orchestra.run.smithi111.stdout:896 write 0x684e7f6 thru 0x68542be (0x5ac9 bytes) 2024-04-23T21:21:31.358 INFO:teuthology.orchestra.run.smithi111.stdout:898 writesame 0xc9f3cf1 thru 0xc9fbd5e data_size 0x11(0x806e bytes) 2024-04-23T21:21:31.362 INFO:teuthology.orchestra.run.smithi111.stdout:899 punch from 0x10eb637 to 0x10f7857, (0xc220 bytes) 2024-04-23T21:21:31.362 INFO:teuthology.orchestra.run.smithi111.stdout:900 writesame 0x910f2c6 thru 0x9115fa7 data_size 0x4d(0x6ce2 bytes) 2024-04-23T21:21:31.374 INFO:teuthology.orchestra.run.smithi111.stdout:902 trunc from 0xe0d406b to 0x6d0896a 2024-04-23T21:21:31.398 INFO:teuthology.orchestra.run.smithi111.stdout:904 writesame 0x966ef56 thru 0x967557e data_size 0x1(0x6629 bytes) 2024-04-23T21:21:31.398 INFO:teuthology.orchestra.run.smithi111.stdout:905 punch from 0x5a9a074 to 0x5aa0360, (0x62ec bytes) 2024-04-23T21:21:31.398 INFO:teuthology.orchestra.run.smithi111.stdout:907 punch from 0x238354c to 0x23890bd, (0x5b71 bytes) 2024-04-23T21:21:31.398 INFO:teuthology.orchestra.run.smithi111.stdout:908 write 0x46af1d2 thru 0x46bea4b (0xf87a bytes) 2024-04-23T21:21:31.405 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.405 INFO:teuthology.orchestra.run.smithi111.stdout:910 punch from 0x28e06f9 to 0x28e9a45, (0x934c bytes) 2024-04-23T21:21:31.405 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.405 INFO:teuthology.orchestra.run.smithi111.stdout:912 punch from 0xb2566c to 0xb2b509, (0x5e9d bytes) 2024-04-23T21:21:31.406 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.406 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.409 INFO:teuthology.orchestra.run.smithi111.stdout:913 trunc from 0x967557f to 0xc7f9065 2024-04-23T21:21:31.414 INFO:teuthology.orchestra.run.smithi111.stdout:914 write 0x42c0cd6 thru 0x42c0e8b (0x1b6 bytes) 2024-04-23T21:21:31.416 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:31.418 INFO:teuthology.orchestra.run.smithi111.stdout:916 write 0x5b56d49 thru 0x5b60a62 (0x9d1a bytes) 2024-04-23T21:21:31.424 INFO:teuthology.orchestra.run.smithi111.stdout:917 read 0xc428cbc thru 0xc432b4f (0x9e94 bytes) 2024-04-23T21:21:31.425 INFO:teuthology.orchestra.run.smithi111.stdout:918 writesame 0x2bb3ad5 thru 0x2bbb0da data_size 0x1(0x7606 bytes) 2024-04-23T21:21:31.440 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:31.440+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:31.442 INFO:teuthology.orchestra.run.smithi111.stdout:919 writesame 0xe1b365a thru 0xe1bbe0e data_size 0x7(0x87b5 bytes) 2024-04-23T21:21:31.442 INFO:teuthology.orchestra.run.smithi111.stdout:920 read 0x1efc0e1 thru 0x1efd7ed (0x170d bytes) 2024-04-23T21:21:31.442 INFO:teuthology.orchestra.run.smithi111.stdout:921 write 0x77dc7e thru 0x78a047 (0xc3ca bytes) 2024-04-23T21:21:31.449 INFO:teuthology.orchestra.run.smithi111.stdout:922 read 0x283cb15 thru 0x284c802 (0xfcee bytes) 2024-04-23T21:21:31.449 INFO:teuthology.orchestra.run.smithi111.stdout:923 writesame 0x2182220 thru 0x218fab5 data_size 0x3(0xd896 bytes) 2024-04-23T21:21:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:926 punch from 0x19021e8 to 0x1909635, (0x744d bytes) 2024-04-23T21:21:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:928 punch from 0x1bde9ea to 0x1be37f9, (0x4e0f bytes) 2024-04-23T21:21:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:929 write 0x96b5a77 thru 0x96b7552 (0x1adc bytes) 2024-04-23T21:21:31.462 INFO:teuthology.orchestra.run.smithi111.stdout:930 write 0x4319906 thru 0x431a679 (0xd74 bytes) 2024-04-23T21:21:31.465 INFO:teuthology.orchestra.run.smithi111.stdout:931 write 0x294b801 thru 0x295b15d (0xf95d bytes) 2024-04-23T21:21:31.470 INFO:teuthology.orchestra.run.smithi111.stdout:932 write 0xc29689e thru 0xc29df0b (0x766e bytes) 2024-04-23T21:21:31.473 INFO:teuthology.orchestra.run.smithi111.stdout:933 writesame 0x72c5c0f thru 0x72ce5b2 data_size 0x2e(0x89a4 bytes) 2024-04-23T21:21:31.476 INFO:teuthology.orchestra.run.smithi111.stdout:935 write 0x4b21525 thru 0x4b2e164 (0xcc40 bytes) 2024-04-23T21:21:31.480 INFO:teuthology.orchestra.run.smithi111.stdout:936 write 0x9209385 thru 0x920d017 (0x3c93 bytes) 2024-04-23T21:21:31.482 INFO:teuthology.orchestra.run.smithi111.stdout:937 write 0x429f56b thru 0x429fed1 (0x967 bytes) 2024-04-23T21:21:31.483 INFO:teuthology.orchestra.run.smithi111.stdout:938 write 0x5f6b6b9 thru 0x5f7b413 (0xfd5b bytes) 2024-04-23T21:21:31.486 INFO:teuthology.orchestra.run.smithi111.stdout:939 punch from 0xc40c0a3 to 0xc41a12e, (0xe08b bytes) 2024-04-23T21:21:31.487 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.487 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.487 INFO:teuthology.orchestra.run.smithi111.stdout:943 write 0x438db69 thru 0x438f447 (0x18df bytes) 2024-04-23T21:21:31.505 INFO:teuthology.orchestra.run.smithi111.stdout:944 trunc from 0xe1bbe0f to 0xc68c71 2024-04-23T21:21:31.537 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.537 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.541 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:31.541+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:31.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:31.550 INFO:teuthology.orchestra.run.smithi111.stdout:945 trunc from 0xc68c71 to 0x94078d7 2024-04-23T21:21:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:946 punch from 0x2697fd5 to 0x26a2c83, (0xacae bytes) 2024-04-23T21:21:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:947 read 0x3f81309 thru 0x3f8d644 (0xc33c bytes) 2024-04-23T21:21:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.555 INFO:teuthology.orchestra.run.smithi111.stdout:952 writesame 0x9dba0a thru 0x9dc468 data_size 0xf(0xa5f bytes) 2024-04-23T21:21:31.556 INFO:teuthology.orchestra.run.smithi111.stdout:953 punch from 0x92c60eb to 0x92cd88e, (0x77a3 bytes) 2024-04-23T21:21:31.556 INFO:teuthology.orchestra.run.smithi111.stdout:954 punch from 0x557cb0a to 0x557d659, (0xb4f bytes) 2024-04-23T21:21:31.557 INFO:teuthology.orchestra.run.smithi111.stdout:956 writesame 0x221d17e thru 0x2224b2e data_size 0x1(0x79b1 bytes) 2024-04-23T21:21:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:958 punch from 0x8020eb2 to 0x802aa5b, (0x9ba9 bytes) 2024-04-23T21:21:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:959 read 0x77b2593 thru 0x77bdebb (0xb929 bytes) 2024-04-23T21:21:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.567 INFO:teuthology.orchestra.run.smithi111.stdout:963 write 0x86d90b3 thru 0x86dfb0b (0x6a59 bytes) 2024-04-23T21:21:31.569 INFO:teuthology.orchestra.run.smithi111.stdout:964 writesame 0x6839633 thru 0x68482d9 data_size 0x1(0xeca7 bytes) 2024-04-23T21:21:31.596 INFO:teuthology.orchestra.run.smithi111.stdout:965 write 0xa2075c1 thru 0xa20c392 (0x4dd2 bytes) 2024-04-23T21:21:31.609 INFO:teuthology.orchestra.run.smithi111.stdout:967 write 0xe4bc4dc thru 0xe4c4971 (0x8496 bytes) 2024-04-23T21:21:31.614 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.614 INFO:teuthology.orchestra.run.smithi111.stdout:970 read 0x79235df thru 0x793066f (0xd091 bytes) 2024-04-23T21:21:31.614 INFO:teuthology.orchestra.run.smithi111.stdout:972 punch from 0x8431e55 to 0x843e809, (0xc9b4 bytes) 2024-04-23T21:21:31.614 INFO:teuthology.orchestra.run.smithi111.stdout:973 write 0x381ecc thru 0x38bdbe (0x9ef3 bytes) 2024-04-23T21:21:31.621 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.621 INFO:teuthology.orchestra.run.smithi111.stdout:977 writesame 0x917b83e thru 0x9182656 data_size 0x5(0x6e19 bytes) 2024-04-23T21:21:31.632 INFO:teuthology.orchestra.run.smithi111.stdout:978 writesame 0xee2b2b0 thru 0xee2c05b data_size 0x5(0xdac bytes) 2024-04-23T21:21:31.632 INFO:teuthology.orchestra.run.smithi111.stdout:979 writesame 0x46fb74b thru 0x46fe9e0 data_size 0x46(0x3296 bytes) 2024-04-23T21:21:31.641 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:31.641+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:31.648 INFO:teuthology.orchestra.run.smithi111.stdout:981 trunc from 0xee2c05c to 0x3842423 2024-04-23T21:21:31.660 INFO:teuthology.orchestra.run.smithi111.stdout:983 trunc from 0x3842423 to 0xfae9d2 2024-04-23T21:21:31.669 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.672 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.672 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.680 INFO:teuthology.orchestra.run.smithi111.stdout:985 write 0x592d4d0 thru 0x593a234 (0xcd65 bytes) 2024-04-23T21:21:31.684 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:31.685 INFO:teuthology.orchestra.run.smithi111.stdout:987 write 0x41ff0f4 thru 0x4206980 (0x788d bytes) 2024-04-23T21:21:31.696 INFO:teuthology.orchestra.run.smithi111.stdout:988 writesame 0x7fcba8d thru 0x7fd7f06 data_size 0xa6(0xc47a bytes) 2024-04-23T21:21:31.701 INFO:teuthology.orchestra.run.smithi111.stdout:989 writesame 0x8404e55 thru 0x8408f21 data_size 0x1(0x40cd bytes) 2024-04-23T21:21:31.701 INFO:teuthology.orchestra.run.smithi111.stdout:991 read 0xbdd718 thru 0xbe3dbe (0x66a7 bytes) 2024-04-23T21:21:31.701 INFO:teuthology.orchestra.run.smithi111.stdout:993 write 0x631c397 thru 0x632adb8 (0xea22 bytes) 2024-04-23T21:21:31.706 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.706 INFO:teuthology.orchestra.run.smithi111.stdout:995 writesame 0x6711737 thru 0x671c9e5 data_size 0x1(0xb2af bytes) 2024-04-23T21:21:31.724 INFO:teuthology.orchestra.run.smithi111.stdout:996 writesame 0x89bf0c thru 0x8a556d data_size 0x1(0x9662 bytes) 2024-04-23T21:21:31.735 INFO:teuthology.orchestra.run.smithi111.stdout:998 read 0x5a48673 thru 0x5a4c795 (0x4123 bytes) 2024-04-23T21:21:31.742 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:31.742+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:31.742 INFO:teuthology.orchestra.run.smithi111.stdout:1000 trunc from 0x8408f22 to 0x2900211 2024-04-23T21:21:31.759 INFO:teuthology.orchestra.run.smithi111.stdout:1001 write 0x4566b80 thru 0x456e487 (0x7908 bytes) 2024-04-23T21:21:31.767 INFO:teuthology.orchestra.run.smithi111.stdout:1002 writesame 0x4d14b79 thru 0x4d24a1b data_size 0x3(0xfea3 bytes) 2024-04-23T21:21:31.768 INFO:teuthology.orchestra.run.smithi111.stdout:1003 read 0xfc00be thru 0xfc8251 (0x8194 bytes) 2024-04-23T21:21:31.768 INFO:teuthology.orchestra.run.smithi111.stdout:1004 writesame 0x326be75 thru 0x3278538 data_size 0x2(0xc6c4 bytes) 2024-04-23T21:21:31.781 INFO:teuthology.orchestra.run.smithi111.stdout:1005 punch from 0x42410b6 to 0x4243a93, (0x29dd bytes) 2024-04-23T21:21:31.781 INFO:teuthology.orchestra.run.smithi111.stdout:1007 read 0xe695c8 thru 0xe738f0 (0xa329 bytes) 2024-04-23T21:21:31.781 INFO:teuthology.orchestra.run.smithi111.stdout:1010 read 0x20df79 thru 0x2183f6 (0xa47e bytes) 2024-04-23T21:21:31.781 INFO:teuthology.orchestra.run.smithi111.stdout:1011 punch from 0x2434591 to 0x243b701, (0x7170 bytes) 2024-04-23T21:21:31.790 INFO:teuthology.orchestra.run.smithi111.stdout:1013 trunc from 0x4d24a1c to 0xc0fbe75 2024-04-23T21:21:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:1014 read 0xe61585 thru 0xe63987 (0x2403 bytes) 2024-04-23T21:21:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:1015 punch from 0x433aeb9 to 0x433e195, (0x32dc bytes) 2024-04-23T21:21:31.795 INFO:teuthology.orchestra.run.smithi111.stdout:1016 read 0x3e420db thru 0x3e465a0 (0x44c6 bytes) 2024-04-23T21:21:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:1017 writesame 0xdf5b044 thru 0xdf612bf data_size 0x2c(0x627c bytes) 2024-04-23T21:21:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:1018 read 0x97514b8 thru 0x9757b09 (0x6652 bytes) 2024-04-23T21:21:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:1019 read 0x5bbdcb3 thru 0x5bcbfd7 (0xe325 bytes) 2024-04-23T21:21:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:1020 punch from 0x3fae07f to 0x3fb40ad, (0x602e bytes) 2024-04-23T21:21:31.803 INFO:teuthology.orchestra.run.smithi111.stdout:1021 writesame 0x4af3c4f thru 0x4afb75f data_size 0x1(0x7b11 bytes) 2024-04-23T21:21:31.812 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.812 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.819 INFO:teuthology.orchestra.run.smithi111.stdout:1022 read 0xbb9c72a thru 0xbb9d632 (0xf09 bytes) 2024-04-23T21:21:31.821 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:31.831 INFO:teuthology.orchestra.run.smithi111.stdout:1024 trunc from 0xdf612c0 to 0x41ca96d 2024-04-23T21:21:31.835 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.0 primary-affinity to 0.0468246 (830682) 2024-04-23T21:21:31.836 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:21:31.843 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:31.843+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:31.851 INFO:teuthology.orchestra.run.smithi111.stdout:1025 writesame 0x1fadd0e thru 0x1fae97a data_size 0x1(0xc6d bytes) 2024-04-23T21:21:31.855 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.855 INFO:teuthology.orchestra.run.smithi111.stdout:1028 punch from 0xea9c31 to 0xeb7d97, (0xe166 bytes) 2024-04-23T21:21:31.856 INFO:teuthology.orchestra.run.smithi111.stdout:1029 punch from 0x12665c2 to 0x1272b7b, (0xc5b9 bytes) 2024-04-23T21:21:31.856 INFO:teuthology.orchestra.run.smithi111.stdout:1030 read 0x2ee1e6c thru 0x2eeb1e0 (0x9375 bytes) 2024-04-23T21:21:31.856 INFO:teuthology.orchestra.run.smithi111.stdout:1035 write 0x11434ff thru 0x114716d (0x3c6f bytes) 2024-04-23T21:21:31.858 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:31.858 INFO:teuthology.orchestra.run.smithi111.stdout:1040 read 0x1b8bf9d thru 0x1b97f14 (0xbf78 bytes) 2024-04-23T21:21:31.858 INFO:teuthology.orchestra.run.smithi111.stdout:1042 read 0x33b7de8 thru 0x33c096a (0x8b83 bytes) 2024-04-23T21:21:31.859 INFO:teuthology.orchestra.run.smithi111.stdout:1043 punch from 0x1f451b to 0x1f7461, (0x2f46 bytes) 2024-04-23T21:21:31.859 INFO:teuthology.orchestra.run.smithi111.stdout:1044 read 0x414ed9c thru 0x415b0c8 (0xc32d bytes) 2024-04-23T21:21:31.865 INFO:teuthology.orchestra.run.smithi111.stdout:1045 write 0x5269783 thru 0x5269ae5 (0x363 bytes) 2024-04-23T21:21:31.876 INFO:teuthology.orchestra.run.smithi111.stdout:1046 write 0x7c741f1 thru 0x7c78773 (0x4583 bytes) 2024-04-23T21:21:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:31.940 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:31.943 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:31.944+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:31.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:32.045 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:32.045+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:32.092 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:32.092 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:32.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:32.185 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:32.185+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:32.243 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:32.244 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:32.245 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:32.246+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:32.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:32.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:32.346 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:32.348+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:32.400 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:32.400 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:32.409 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:32.414 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:32.414 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:32.414 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:32.432 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:32.433 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:32.447 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:32.448+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:32.449 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:32.450 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:32.467 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:32.468 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:32.485 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:32.485 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:32.502 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:32.502 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:32.512 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:32.513 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:32.513 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:32.513 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:32.513 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:32.520 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:32.520 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:32.522 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:32.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:32.548 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:32.548+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:32.648 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:32.650+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7260 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6684672~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2153362919999999, 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026061528, 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7260 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451672+0000", 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3179999999999999e-06 2024-04-23T21:21:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451676+0000", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4199999999999999e-06 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451680+0000", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6279999999999998e-06 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477564+0000", 2024-04-23T21:21:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025884564999999998 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477641+0000", 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7300999999999997e-05 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477732+0000", 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0884999999999995e-05 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7261 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6881280~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2153221219999999, 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026239373, 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7261 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5100000000000001e-07 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451688+0000", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.255e-06 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451693+0000", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8540000000000002e-06 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451698+0000", 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4610000000000001e-06 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477761+0000", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026062980999999999 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477836+0000", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5496999999999999e-05 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477924+0000", 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7973999999999995e-05 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7262 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7077888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2153042169999999, 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026422265, 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7262 2024-04-23T21:21:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9200000000000002e-07 2024-04-23T21:21:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451705+0000", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8250000000000001e-06 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451712+0000", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.759e-06 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451716+0000", 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2889999999999999e-06 2024-04-23T21:21:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477952+0000", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026235967999999998 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478032+0000", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0296999999999994e-05 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478125+0000", 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2734999999999999e-05 2024-04-23T21:21:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7263 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7798784~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2152834850000001, 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026605936, 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7263 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452459+0000", 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00073525200000000004 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452464+0000", 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8940000000000001e-06 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452474+0000", 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0454e-05 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452481+0000", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7469999999999999e-06 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478153+0000", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025672186 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478236+0000", 2024-04-23T21:21:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3061999999999999e-05 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478329+0000", 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3341e-05 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7265 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8192000~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.214493568, 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026201490000000001, 2024-04-23T21:21:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7265 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452514+0000", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7899999999999999e-07 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452516+0000", 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452522+0000", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.088e-06 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452527+0000", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9420000000000003e-06 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478555+0000", 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026027557 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478630+0000", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4892000000000001e-05 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478715+0000", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5436e-05 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7266 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8388608~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2144741469999998, 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026420895999999999, 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7266 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452535+0000", 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7990000000000001e-06 2024-04-23T21:21:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452541+0000", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4670000000000003e-06 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452545+0000", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2089999999999999e-06 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478797+0000", 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026252622 2024-04-23T21:21:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478883+0000", 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5390999999999996e-05 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478954+0000", 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1044000000000006e-05 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7267 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9109504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2144554460000001, 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026543255000000002, 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7267 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.41e-07 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452554+0000", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.041e-06 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452560+0000", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1179999999999997e-06 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452565+0000", 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5020000000000003e-06 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478989+0000", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026423965000000001 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479040+0000", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1697000000000003e-05 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479095+0000", 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4591e-05 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7268 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9306112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.21443703, 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026804591999999999, 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7268 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452572+0000", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8419999999999999e-06 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452578+0000", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3550000000000004e-06 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452582+0000", 2024-04-23T21:21:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2780000000000004e-06 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479186+0000", 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026603907 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479275+0000", 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8918999999999993e-05 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479375+0000", 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9942e-05 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7269 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9502720~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2144191590000002, 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026998115, 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7269 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452611+0000", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2425e-05 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452617+0000", 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7580000000000003e-06 2024-04-23T21:21:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452624+0000", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7109999999999998e-06 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479405+0000", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026780531999999999 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479498+0000", 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3045000000000001e-05 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479586+0000", 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8288999999999994e-05 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7270 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9699328~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2143753099999999, 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027156907000000001, 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7270 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452635+0000", 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.187e-06 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452640+0000", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7910000000000001e-06 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452645+0000", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8289999999999998e-06 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479616+0000", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026971222 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479697+0000", 2024-04-23T21:21:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0463000000000005e-05 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479789+0000", 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1810000000000004e-05 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7271 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10420224~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.214354937, 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027566067, 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7271 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452653+0000", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452655+0000", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0559999999999999e-06 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452660+0000", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6289999999999998e-06 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452665+0000", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6430000000000003e-06 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480043+0000", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027378070000000001 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480130+0000", 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6879999999999995e-05 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480218+0000", 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8357999999999997e-05 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7388 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6717440~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2083493390000002, 2024-04-23T21:21:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026122209, 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7388 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5200000000000002e-07 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458660+0000", 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5719999999999999e-06 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458665+0000", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6609999999999997e-06 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458670+0000", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3889999999999998e-06 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484666+0000", 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025996717999999999 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484724+0000", 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7222999999999997e-05 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484780+0000", 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6193999999999998e-05 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7389 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6914048~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.208331807, 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026331738, 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7389 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458676+0000", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7500000000000001e-07 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458678+0000", 2024-04-23T21:21:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6970000000000002e-06 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458684+0000", 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5709999999999997e-06 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458688+0000", 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6099999999999999e-06 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484821+0000", 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026132275 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484908+0000", 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7822e-05 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485007+0000", 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8487999999999996e-05 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7390 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7110656~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2083130940000002, 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026475571, 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7390 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458702+0000", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458708+0000", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0100000000000001e-06 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458712+0000", 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:21:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485062+0000", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026349392999999999 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485112+0000", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0466000000000001e-05 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485170+0000", 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7160999999999997e-05 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7391 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7831552~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.208288504, 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026684876999999999, 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7391 2024-04-23T21:21:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7800000000000002e-07 2024-04-23T21:21:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458722+0000", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.587e-06 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458728+0000", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5189999999999998e-06 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458733+0000", 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7140000000000001e-06 2024-04-23T21:21:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485207+0000", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026474393999999998 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485300+0000", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2644000000000003e-05 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485403+0000", 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010364100000000001 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7392 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8028160~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2082655600000001, 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026904931999999999, 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7392 2024-04-23T21:21:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458742+0000", 2024-04-23T21:21:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5999999999999999e-07 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458744+0000", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8509999999999999e-06 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458749+0000", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6119999999999998e-06 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458755+0000", 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1440000000000002e-06 2024-04-23T21:21:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485459+0000", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026704828 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485548+0000", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8985000000000003e-05 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485646+0000", 2024-04-23T21:21:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8052000000000002e-05 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7393 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8224768~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2082468049999999, 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027031339000000001, 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7393 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460437+0000", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001677053 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460442+0000", 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460454+0000", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1704e-05 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460461+0000", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4909999999999999e-06 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485685+0000", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025223992000000001 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485736+0000", 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0633999999999997e-05 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485792+0000", 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5653999999999999e-05 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7397 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9535488~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2064412949999999, 2024-04-23T21:21:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026115167000000002, 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7397 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460568+0000", 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1459999999999999e-06 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460574+0000", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6720000000000001e-06 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460579+0000", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486493+0000", 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025914311999999998 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486582+0000", 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9132e-05 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486681+0000", 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8956000000000006e-05 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.742 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7398 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9732096~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.206422377, 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026253736, 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7398 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:32.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460587+0000", 2024-04-23T21:21:32.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.035e-06 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460593+0000", 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6670000000000001e-06 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460597+0000", 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4789999999999998e-06 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486723+0000", 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026126018000000001 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486779+0000", 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5445000000000002e-05 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486838+0000", 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9704000000000003e-05 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7399 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10452992~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.2064041259999998, 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026351192999999998, 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7399 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5699999999999998e-07 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460605+0000", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.781e-06 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460611+0000", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5509999999999999e-06 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460615+0000", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5619999999999997e-06 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:32.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486864+0000", 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026248947000000002 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486911+0000", 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-05 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486954+0000", 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2664999999999999e-05 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.750 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:32.751 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:32.751 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:32.752 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:32.753 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:32.753+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:32.845 INFO:teuthology.orchestra.run.smithi111.stdout:1048 read 0x1ef2d1b thru 0x1f025c4 (0xf8aa bytes) 2024-04-23T21:21:32.846 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.850 INFO:teuthology.orchestra.run.smithi111.stdout:1055 trunc from 0x7c78774 to 0xbd22b9b 2024-04-23T21:21:32.852 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:32.853+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:32.857 INFO:teuthology.orchestra.run.smithi111.stdout:1056 write 0x951d0ec thru 0x951f603 (0x2518 bytes) 2024-04-23T21:21:32.861 INFO:teuthology.orchestra.run.smithi111.stdout:1057 read 0x71a5c97 thru 0x71aeefb (0x9265 bytes) 2024-04-23T21:21:32.861 INFO:teuthology.orchestra.run.smithi111.stdout:1058 punch from 0x38dba73 to 0x38e52c1, (0x984e bytes) 2024-04-23T21:21:32.861 INFO:teuthology.orchestra.run.smithi111.stdout:1060 write 0x6a372f9 thru 0x6a44b3d (0xd845 bytes) 2024-04-23T21:21:32.864 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:32.864 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:32.864 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:32.864 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:32.867 INFO:teuthology.orchestra.run.smithi111.stdout:1061 punch from 0x22589dc to 0x2258a9c, (0xc0 bytes) 2024-04-23T21:21:32.867 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.867 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.867 INFO:teuthology.orchestra.run.smithi111.stdout:1065 read 0x5f7720d thru 0x5f7d45a (0x624e bytes) 2024-04-23T21:21:32.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:32.878 INFO:teuthology.orchestra.run.smithi111.stdout:1066 trunc from 0xbd22b9b to 0x1fc3f1c 2024-04-23T21:21:32.891 INFO:teuthology.orchestra.run.smithi111.stdout:1067 read 0xea4b76 thru 0xeaa094 (0x551f bytes) 2024-04-23T21:21:32.891 INFO:teuthology.orchestra.run.smithi111.stdout:1068 read 0x8c1867 thru 0x8ca1e9 (0x8983 bytes) 2024-04-23T21:21:32.891 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.904 INFO:teuthology.orchestra.run.smithi111.stdout:1070 trunc from 0x1fc3f1c to 0xc79322b 2024-04-23T21:21:32.909 INFO:teuthology.orchestra.run.smithi111.stdout:1072 read 0xb9e946a thru 0xb9f8d3b (0xf8d2 bytes) 2024-04-23T21:21:32.909 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.910 INFO:teuthology.orchestra.run.smithi111.stdout:1075 write 0x79f0673 thru 0x79ff4c9 (0xee57 bytes) 2024-04-23T21:21:32.925 INFO:teuthology.orchestra.run.smithi111.stdout:1076 trunc from 0xc79322b to 0x244b6d1 2024-04-23T21:21:32.953 INFO:teuthology.orchestra.run.smithi111.stdout:1077 write 0xc82a1c0 thru 0xc8372a0 (0xd0e1 bytes) 2024-04-23T21:21:32.957 INFO:teuthology.orchestra.run.smithi111.stdout:1079 write 0xc812162 thru 0xc81e482 (0xc321 bytes) 2024-04-23T21:21:32.958 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:32.959+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:32.964 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:32.965 INFO:teuthology.orchestra.run.smithi111.stdout:1081 trunc from 0xc8372a1 to 0xdba3436 2024-04-23T21:21:32.971 INFO:teuthology.orchestra.run.smithi111.stdout:1083 read 0x8dbf6e5 thru 0x8dbfd6c (0x688 bytes) 2024-04-23T21:21:32.971 INFO:teuthology.orchestra.run.smithi111.stdout:1085 write 0x2b2316f thru 0x2b25ed6 (0x2d68 bytes) 2024-04-23T21:21:32.973 INFO:teuthology.orchestra.run.smithi111.stdout:1086 punch from 0xc234bf5 to 0xc23a73a, (0x5b45 bytes) 2024-04-23T21:21:32.974 INFO:teuthology.orchestra.run.smithi111.stdout:1087 read 0x60b4da2 thru 0x60bdb09 (0x8d68 bytes) 2024-04-23T21:21:32.974 INFO:teuthology.orchestra.run.smithi111.stdout:1090 write 0x7dfed42 thru 0x7e06d22 (0x7fe1 bytes) 2024-04-23T21:21:32.980 INFO:teuthology.orchestra.run.smithi111.stdout:1091 clone 8 order 25 su 1048576 sc 15 2024-04-23T21:21:33.019 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:33.020 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:33.069 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:33.070+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:33.155 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:33.156+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:33.171 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3136 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3080192~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7545304850000001, 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013839865, 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3136 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:33.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417368+0000", 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417370+0000", 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417377+0000", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6030000000000001e-06 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417381+0000", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3030000000000001e-06 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431018+0000", 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013636540000000001 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431101+0000", 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3014000000000002e-05 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431208+0000", 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010699199999999999 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.176 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3137 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3145728~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7545109760000006, 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014041269, 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3137 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:33.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417394+0000", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.55e-06 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417401+0000", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5930000000000002e-06 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417406+0000", 2024-04-23T21:21:33.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5059999999999997e-06 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431251+0000", 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013845245000000001 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431327+0000", 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6409000000000007e-05 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431429+0000", 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101582 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3138 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3211264~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7544864980000003, 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014240901, 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3138 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8099999999999998e-07 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417419+0000", 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8979999999999999e-06 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417426+0000", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5100000000000004e-06 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417431+0000", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7940000000000002e-06 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431471+0000", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014040336 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431551+0000", 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0343e-05 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431653+0000", 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010163899999999999 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3139 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3276800~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7544615449999998, 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014372180999999999, 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3139 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417439+0000", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8950000000000001e-06 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417453+0000", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3383e-05 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417458+0000", 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:21:33.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431688+0000", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014230456000000001 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431755+0000", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7183999999999999e-05 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431809+0000", 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3816000000000001e-05 2024-04-23T21:21:33.189 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3140 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3342336~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7544337280000004, 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014584774999999999, 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3140 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417467+0000", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417473+0000", 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2559999999999999e-06 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417478+0000", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3510000000000004e-06 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431863+0000", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014384868 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431944+0000", 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1433000000000005e-05 2024-04-23T21:21:33.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432050+0000", 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105503 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3141 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7544116570000003, 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014710842999999999, 2024-04-23T21:21:33.194 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3141 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417489+0000", 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7269999999999999e-06 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417495+0000", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2060000000000004e-06 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417506+0000", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1277000000000001e-05 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432144+0000", 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014637281 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432198+0000", 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3965999999999998e-05 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3142 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7543851799999999, 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014717164, 2024-04-23T21:21:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3142 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417514+0000", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417516+0000", 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.875e-06 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417522+0000", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4169999999999997e-06 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417526+0000", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4410000000000003e-06 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432223+0000", 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014696300000000001 2024-04-23T21:21:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432231+0000", 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7409999999999992e-06 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3143 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7543656769999991, 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730144000000001, 2024-04-23T21:21:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3143 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:33.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:33.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417535+0000", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7090000000000001e-06 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417541+0000", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2110000000000003e-06 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417551+0000", 2024-04-23T21:21:33.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9250000000000007e-06 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432251+0000", 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014700210999999999 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432263+0000", 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1715999999999999e-05 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3144 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.754340827, 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014732579000000001, 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3144 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.46e-07 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417560+0000", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.945e-06 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417566+0000", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1909999999999997e-06 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417571+0000", 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432283+0000", 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014711864999999999 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432290+0000", 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.525e-06 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3145 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.211 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7543214620000001, 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014738942, 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3145 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:33.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417580+0000", 2024-04-23T21:21:33.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.125e-06 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417586+0000", 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3339999999999998e-06 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417591+0000", 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6809999999999997e-06 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432309+0000", 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014718916 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432316+0000", 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3146 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.215 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7542941479999996, 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014741944, 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3146 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417605+0000", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417607+0000", 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417612+0000", 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5079999999999996e-06 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417617+0000", 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1919999999999998e-06 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432340+0000", 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014723234 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432346+0000", 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4910000000000002e-06 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3147 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.219 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7542752959999994, 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014787737, 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3147 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417624+0000", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417626+0000", 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.012e-06 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417632+0000", 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0220000000000001e-06 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417636+0000", 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3000000000000003e-06 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432366+0000", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730115 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432411+0000", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4895000000000003e-05 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3148 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:33.223 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7542566429999997, 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014811695999999999, 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3148 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6100000000000002e-07 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417650+0000", 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1019999999999997e-06 2024-04-23T21:21:33.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417658+0000", 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1189999999999997e-06 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417664+0000", 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432443+0000", 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014779336000000001 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432454+0000", 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0713999999999999e-05 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3149 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7542277770000005, 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014815238, 2024-04-23T21:21:33.227 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3149 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3500000000000002e-07 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417673+0000", 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417679+0000", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4530000000000001e-06 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417683+0000", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2560000000000004e-06 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432479+0000", 2024-04-23T21:21:33.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014796236000000001 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432486+0000", 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7479999999999996e-06 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3150 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.754209328, 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014825331000000001, 2024-04-23T21:21:33.231 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3150 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417690+0000", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1600000000000001e-07 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417696+0000", 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3169999999999998e-06 2024-04-23T21:21:33.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417703+0000", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1729999999999998e-06 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417708+0000", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.481e-06 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432508+0000", 2024-04-23T21:21:33.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014799827999999999 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432515+0000", 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160000000000003e-06 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3151 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7541843440000005, 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014864755, 2024-04-23T21:21:33.235 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3151 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417715+0000", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417717+0000", 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2500000000000001e-06 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417724+0000", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.289e-06 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417729+0000", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650000000000002e-06 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432567+0000", 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014837845000000001 2024-04-23T21:21:33.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432579+0000", 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2334000000000001e-05 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3152 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7541633700000006, 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014901721, 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.239 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3152 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417736+0000", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.96e-07 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417738+0000", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0540000000000002e-06 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417743+0000", 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2510000000000002e-06 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417755+0000", 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2536e-05 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432629+0000", 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014873569 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432637+0000", 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9149999999999996e-06 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:4908 3.3 3:c0665d5c:::rbd_data.11181344e3fc.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 11010048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.9848946060000001, 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030900347000000002, 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.243 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4908 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:33.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187005+0000", 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9800000000000002e-07 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198009+0000", 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011004491999999999 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198032+0000", 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2753000000000001e-05 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198059+0000", 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6792e-05 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198122+0000", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3633999999999994e-05 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198236+0000", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011307 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.199108+0000", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087266500000000001 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.210413+0000", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011304611000000001 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217815+0000", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0074019869999999996 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217856+0000", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1411e-05 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217904+0000", 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8034000000000001e-05 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4935 3.1b e26/18 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:head v 26'8, mlcod=24'5)", 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.9289425409999996, 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024520146999999999, 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242957+0000", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7800000000000001e-07 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259689+0000", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016732883 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259714+0000", 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4992000000000001e-05 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259726+0000", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1916000000000001e-05 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259814+0000", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7964e-05 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259834+0000", 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0083999999999999e-05 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267434+0000", 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0075992239999999999 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267476+0000", 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2506000000000003e-05 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4953 3.1b e26/18 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:head v 26'9, mlcod=25'7)", 2024-04-23T21:21:33.253 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.8921228980000002, 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024344686000000001, 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279777+0000", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.75e-06 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295820+0000", 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016042178000000001 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295845+0000", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5225e-05 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295854+0000", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.064e-06 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295951+0000", 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6806999999999997e-05 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295969+0000", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8672999999999999e-05 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304074+0000", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081043190000000005 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304120+0000", 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6669999999999999e-05 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.257 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.258 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.258 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.258 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.258 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:33.260 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:33.260+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:33.360 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:33.361+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:33.363 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.363 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:33.363 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:33.363 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:33.461 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:33.461+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:33.514 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:33.514 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:33.566 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:33.567+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2140 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7372800~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.627 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3446847650000002, 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032102877000000002, 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2140 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:33.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282177+0000", 2024-04-23T21:21:33.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0379999999999998e-06 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282185+0000", 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9119999999999998e-06 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282190+0000", 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7269999999999998e-06 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314126+0000", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031936101000000001 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314204+0000", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7791000000000001e-05 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314278+0000", 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3696999999999996e-05 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3446587710000006, 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.632 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:33.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:21:33.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:21:33.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3446386930000003, 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:21:33.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:21:33.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3446180640000005, 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:21:33.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:21:33.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:21:33.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3445928059999996, 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:21:33.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:33.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3445662269999996, 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.650 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3445460730000001, 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3445203130000003, 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:21:33.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444890239999996, 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:21:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:33.667 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444744570000005, 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:21:33.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:21:33.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.671 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444550409999998, 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:33.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.675 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444398579999994, 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.676 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:21:33.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444292890000007, 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:21:33.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:21:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444196720000008, 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:21:33.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.3444101400000008, 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:21:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.344397142, 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:21:33.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:21:33.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.217351184, 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:21:33.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:21:33.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.2173324379999997, 2024-04-23T21:21:33.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:21:33.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.2172964470000007, 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:21:33.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:21:33.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.2172688649999994, 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:33.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:33.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:33.716 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:33.718 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:33.717+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:33.819 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:33.819+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:33.854 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:21:33.864 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:33.865 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:33.917 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone7 from 0xdba3436 (overlap 0xc68c71) to 0x0 2024-04-23T21:21:33.919 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:33.920+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:33.990 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:33.991 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:33.991 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:33.991 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:33.991 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:33.991 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-04-23T21:21:33.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:34.020 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:34.020+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:34.121 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:34.121+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:1633 3.a 3:535385f0:::rbd_children:head [call rbd.add_child in=12b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:34.145 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.29407992, 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023809069999999998, 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 1633 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850632+0000", 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850633+0000", 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4160000000000001e-06 2024-04-23T21:21:34.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850640+0000", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6919999999999996e-06 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850654+0000", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4722000000000001e-05 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850661+0000", 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8059999999999997e-06 2024-04-23T21:21:34.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850710+0000", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9081e-05 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.850714+0000", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8650000000000003e-06 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852078+0000", 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013635400000000001 2024-04-23T21:21:34.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852122+0000", 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4078e-05 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852245+0000", 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012301 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852783+0000", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053843000000000001 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852956+0000", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017254799999999999 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.852989+0000", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2802000000000002e-05 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:23.853013+0000", 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3917000000000002e-05 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:2565 3.1e e23/18 3:7adf3bd0:::rbd_data.11183c92f5c.0000000000000005:head v 23'23, mlcod=22'21)", 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.7737078400000001, 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063423730000000001, 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.74e-07 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374893+0000", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038888899999999999 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374909+0000", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374921+0000", 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2031000000000001e-05 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374987+0000", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5827000000000005e-05 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.375003+0000", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6393e-05 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377299+0000", 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002295503 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377346+0000", 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7475000000000002e-05 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:4868 3.a 3:51bf0a15:::rbd_header.11181344e3fc:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.145165+0000", 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.9995464819999995, 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025999410000000001, 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4868 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145165+0000", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145165+0000", 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145166+0000", 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9999999999999996e-07 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145170+0000", 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0659999999999997e-06 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145180+0000", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0059e-05 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145185+0000", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0749999999999997e-06 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145231+0000", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6091999999999998e-05 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145283+0000", 2024-04-23T21:21:34.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1925000000000002e-05 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145417+0000", 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013347599999999999 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.145781+0000", 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036433499999999997 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.147708+0000", 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019266610000000001 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.147739+0000", 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-05 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.147765+0000", 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5952000000000002e-05 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:5067 3.a 3:51bf0a15:::rbd_header.11181344e3fc:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.426763+0000", 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.7179483759999998, 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002553704, 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.162 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 5067 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426763+0000", 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426763+0000", 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426764+0000", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1800000000000005e-07 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426768+0000", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.014e-06 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426778+0000", 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.842e-06 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426783+0000", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1429999999999997e-06 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426832+0000", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9199000000000001e-05 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.426915+0000", 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3070000000000003e-05 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.427137+0000", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022142900000000001 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.427531+0000", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039444300000000002 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.429221+0000", 2024-04-23T21:21:34.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001689685 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.429272+0000", 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1468999999999998e-05 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.429317+0000", 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4592000000000003e-05 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.167 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:5141 3.a 3:51bf0a15:::rbd_header.11181344e3fc:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.523210+0000", 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.6215011659999998, 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002611557, 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 5141 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523210+0000", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523210+0000", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523211+0000", 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7700000000000004e-07 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523215+0000", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8360000000000002e-06 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523225+0000", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0025e-05 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523230+0000", 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:21:34.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523273+0000", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2784000000000002e-05 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523320+0000", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6987999999999998e-05 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.523449+0000", 2024-04-23T21:21:34.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000128823 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.525605+0000", 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021557120000000002 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.525769+0000", 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016462599999999999 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.525799+0000", 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0074e-05 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.525822+0000", 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2742e-05 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:5235 3.1e e26/18 3:7ca50bd1:::rbd_data.11181344e3fc.000000000000000e:head v 26'44, mlcod=26'42)", 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.631609+0000", 2024-04-23T21:21:34.173 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.5131030540000001, 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021588150000000001, 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631609+0000", 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631609+0000", 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631609+0000", 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1800000000000005e-07 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631615+0000", 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0430000000000004e-06 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631627+0000", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2031999999999999e-05 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631635+0000", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2980000000000003e-06 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631675+0000", 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0491999999999998e-05 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.631688+0000", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3219e-05 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.633721+0000", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020324499999999999 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.633767+0000", 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6462999999999999e-05 2024-04-23T21:21:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:5262 3.a 3:51bf0a15:::rbd_header.11181344e3fc:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.824877+0000", 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3198344369999999, 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002415642, 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 5262 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824877+0000", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824877+0000", 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824879+0000", 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3960000000000001e-06 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824885+0000", 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2530000000000001e-06 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824898+0000", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3056e-05 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824905+0000", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2910000000000002e-06 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824962+0000", 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7028999999999999e-05 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.824966+0000", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9500000000000003e-06 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.826233+0000", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001266744 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.826298+0000", 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5531000000000007e-05 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.826521+0000", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022231099999999999 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.827045+0000", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00052417700000000002 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.827222+0000", 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017702899999999999 2024-04-23T21:21:34.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.827254+0000", 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2237999999999999e-05 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.827293+0000", 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8637000000000003e-05 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.184 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7401 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.6993496329999997, 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036060879999999999, 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7401 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445363+0000", 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7000000000000003e-07 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445969+0000", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060587 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445981+0000", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1731e-05 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445986+0000", 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-04-23T21:21:34.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446040+0000", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3427000000000002e-05 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446176+0000", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013631200000000001 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446400+0000", 2024-04-23T21:21:34.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022467900000000001 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.447547+0000", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001146527 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448901+0000", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013539859999999999 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448939+0000", 2024-04-23T21:21:34.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7608000000000003e-05 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448968+0000", 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9410000000000001e-05 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7418 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.531621+0000", 2024-04-23T21:21:34.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.6130903129999998, 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024680840000000002, 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7418 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531621+0000", 2024-04-23T21:21:34.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531621+0000", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531623+0000", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9999999999999999e-06 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531632+0000", 2024-04-23T21:21:34.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531648+0000", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6540000000000001e-05 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531655+0000", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7519999999999999e-06 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531716+0000", 2024-04-23T21:21:34.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0612000000000002e-05 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.531804+0000", 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8028000000000005e-05 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.532048+0000", 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024401199999999999 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.533817+0000", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017690869999999999 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.534034+0000", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000217381 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.534064+0000", 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0294000000000002e-05 2024-04-23T21:21:34.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.534089+0000", 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4899999999999999e-05 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7476 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [stat,write 3276800~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.5300870900000003, 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039134579999999999, 2024-04-23T21:21:34.196 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7476 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614627+0000", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4770000000000002e-06 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614667+0000", 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9662000000000001e-05 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614687+0000", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0811000000000001e-05 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614698+0000", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0465000000000001e-05 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614755+0000", 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6702e-05 2024-04-23T21:21:34.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614809+0000", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4886000000000003e-05 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614900+0000", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0829999999999996e-05 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.615705+0000", 2024-04-23T21:21:34.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080430599999999997 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618432+0000", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027271420000000001 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618490+0000", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8304000000000003e-05 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618538+0000", 2024-04-23T21:21:34.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7873000000000001e-05 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7498 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [stat,truncate 3768320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.5045084380000002, 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050186079999999999, 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7498 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640204+0000", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1319999999999999e-06 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640209+0000", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0420000000000002e-06 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640223+0000", 2024-04-23T21:21:34.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3813e-05 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640232+0000", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.473e-06 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640292+0000", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0401e-05 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640374+0000", 2024-04-23T21:21:34.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1729000000000003e-05 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640503+0000", 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129348 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.641060+0000", 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00055648700000000004 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645148+0000", 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004088688 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645184+0000", 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5491000000000003e-05 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645222+0000", 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8003999999999997e-05 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7650 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [stat,truncate 3047424] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.812989+0000", 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.331723019, 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024700579999999998, 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7650 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.208 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.812989+0000", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.812989+0000", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.812991+0000", 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1569999999999998e-06 2024-04-23T21:21:34.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813005+0000", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4039000000000001e-05 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813027+0000", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1990000000000001e-05 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813037+0000", 2024-04-23T21:21:34.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9429999999999995e-06 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813140+0000", 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102902 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813227+0000", 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7663000000000004e-05 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:34.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.813360+0000", 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013299899999999999 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.814706+0000", 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013459500000000001 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.815370+0000", 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00066408999999999999 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.815416+0000", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5859000000000003e-05 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.815459+0000", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2466000000000003e-05 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7660 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.819030+0000", 2024-04-23T21:21:34.213 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.3256815910000004, 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0020841800000000001, 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7660 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819030+0000", 2024-04-23T21:21:34.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819030+0000", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819031+0000", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1909999999999999e-06 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819036+0000", 2024-04-23T21:21:34.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2079999999999999e-06 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819049+0000", 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2343e-05 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819055+0000", 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.93e-06 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819103+0000", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8844000000000002e-05 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819165+0000", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1420999999999994e-05 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.819296+0000", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000131571 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.820910+0000", 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016137479999999999 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.821044+0000", 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013382499999999999 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.821080+0000", 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5852000000000002e-05 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.821114+0000", 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4246999999999999e-05 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7866 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.084499+0000", 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0602123990000001, 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0024491669999999999, 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.219 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7866 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084499+0000", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084499+0000", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084501+0000", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8390000000000001e-06 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084507+0000", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2079999999999997e-06 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084522+0000", 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4612e-05 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084529+0000", 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0280000000000003e-06 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084575+0000", 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6544e-05 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084628+0000", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2775000000000002e-05 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.084756+0000", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012764699999999999 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.085209+0000", 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000453296 2024-04-23T21:21:34.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.086883+0000", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016734370000000001 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.086913+0000", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0837999999999998e-05 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.086948+0000", 2024-04-23T21:21:34.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4943000000000001e-05 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7897 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.115705+0000", 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0290066070000004, 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002201728, 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7897 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115705+0000", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115705+0000", 2024-04-23T21:21:34.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115706+0000", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3030000000000001e-06 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115712+0000", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0379999999999996e-06 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115727+0000", 2024-04-23T21:21:34.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4309e-05 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115733+0000", 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8369999999999999e-06 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115782+0000", 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8105000000000001e-05 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115836+0000", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3953999999999999e-05 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.115954+0000", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0001184 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.117662+0000", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001708298 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.117848+0000", 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018621800000000001 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.117876+0000", 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8041000000000001e-05 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.117907+0000", 2024-04-23T21:21:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0224999999999999e-05 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7914 3.10 3:0dec05f2:::rbd_data.111830ac55a1.000000000000001e:head [set-alloc-hint object_size 4194304 write_size 4194304,write 302178~25502] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.130687+0000", 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.0140245490000002, 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021742810000000001, 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.234 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7914 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130687+0000", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130687+0000", 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130688+0000", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1400000000000001e-06 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130706+0000", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7584e-05 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130721+0000", 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.518e-05 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130730+0000", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6989999999999992e-06 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130777+0000", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7207999999999999e-05 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130822+0000", 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5107e-05 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.130903+0000", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0819e-05 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.131466+0000", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056322900000000005 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.132807+0000", 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001341098 2024-04-23T21:21:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.132838+0000", 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0725999999999997e-05 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.132861+0000", 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3490999999999999e-05 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7999 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.208012+0000", 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 3.9366997330000002, 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023521010000000001, 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7999 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208012+0000", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208012+0000", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208013+0000", 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3650000000000001e-06 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208020+0000", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7730000000000002e-06 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208035+0000", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4732000000000001e-05 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208043+0000", 2024-04-23T21:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3489999999999992e-06 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208093+0000", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0380000000000002e-05 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208159+0000", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5833000000000005e-05 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208293+0000", 2024-04-23T21:21:34.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013374200000000001 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.208785+0000", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00049169199999999995 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.210306+0000", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001521564 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.210338+0000", 2024-04-23T21:21:34.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1491000000000001e-05 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.210364+0000", 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.618e-05 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:8046 3.a 3:56ba5b24:::rbd_data.111830ac55a1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 3.9017660350000001, 2024-04-23T21:21:34.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014462938, 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 8046 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242952+0000", 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4650000000000002e-06 2024-04-23T21:21:34.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242966+0000", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3708e-05 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242974+0000", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0369999999999995e-06 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243016+0000", 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2769000000000003e-05 2024-04-23T21:21:34.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243063+0000", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999997e-05 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.245737+0000", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026734939999999998 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257269+0000", 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011532351 2024-04-23T21:21:34.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257289+0000", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0605e-05 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257352+0000", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2535999999999999e-05 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257408+0000", 2024-04-23T21:21:34.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6450000000000003e-05 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:8046 3.a e27/18 ondisk, result = 0)", 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 3.891412581, 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003980267, 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253300+0000", 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5099999999999998e-07 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253303+0000", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5279999999999999e-06 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253315+0000", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1365999999999999e-05 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253324+0000", 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9949999999999994e-06 2024-04-23T21:21:34.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257249+0000", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003925057 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257261+0000", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2340999999999999e-05 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257279+0000", 2024-04-23T21:21:34.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8128999999999999e-05 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10129 3.a 3:547b3703:::rbd_header.111830ac55a1:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:33.879976+0000", 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.26473575700000002, 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021501409999999999, 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10129 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.879976+0000", 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.879976+0000", 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.879978+0000", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9e-06 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.879986+0000", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5609999999999999e-06 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.880006+0000", 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9551e-05 2024-04-23T21:21:34.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.880016+0000", 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8679999999999994e-06 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.880072+0000", 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5943999999999998e-05 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op must wait for map", 2024-04-23T21:21:34.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.880076+0000", 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2130000000000002e-06 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.881946+0000", 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001869899 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.881996+0000", 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9836000000000002e-05 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:33.882126+0000", 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000130369 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.260 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:34.261 INFO:teuthology.orchestra.run.smithi111.stdout:1093 trunc from 0xdba3436 to 0x6c55c5a 2024-04-23T21:21:34.261 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.261 INFO:teuthology.orchestra.run.smithi111.stdout:1095 read 0x97c77d thru 0x986de3 (0xa667 bytes) 2024-04-23T21:21:34.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:34.262 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:34.262+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:34.268 INFO:teuthology.orchestra.run.smithi111.stdout:1096 writesame 0xc08e88b thru 0xc09cba5 data_size 0x1(0xe31b bytes) 2024-04-23T21:21:34.278 INFO:teuthology.orchestra.run.smithi111.stdout:1100 write 0xe72363a thru 0xe72d0e4 (0x9aab bytes) 2024-04-23T21:21:34.279 INFO:teuthology.orchestra.run.smithi111.stdout:1102 punch from 0xb3adf3d to 0xb3b696b, (0x8a2e bytes) 2024-04-23T21:21:34.279 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.279 INFO:teuthology.orchestra.run.smithi111.stdout:1105 writesame 0x5eec2e1 thru 0x5ef2ae6 data_size 0x5(0x6806 bytes) 2024-04-23T21:21:34.282 INFO:teuthology.orchestra.run.smithi111.stdout:1106 write 0xdf20890 thru 0xdf2cd2c (0xc49d bytes) 2024-04-23T21:21:34.289 INFO:teuthology.orchestra.run.smithi111.stdout:1108 trunc from 0xe72d0e5 to 0xa69edb6 2024-04-23T21:21:34.304 INFO:teuthology.orchestra.run.smithi111.stdout:1109 writesame 0xc659fe1 thru 0xc6631a8 data_size 0x2(0x91c8 bytes) 2024-04-23T21:21:34.305 INFO:teuthology.orchestra.run.smithi111.stdout:1110 read 0xcc4c13 thru 0xcc99f5 (0x4de3 bytes) 2024-04-23T21:21:34.305 INFO:teuthology.orchestra.run.smithi111.stdout:1111 read 0x7b949c8 thru 0x7ba1697 (0xccd0 bytes) 2024-04-23T21:21:34.305 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.305 INFO:teuthology.orchestra.run.smithi111.stdout:1113 punch from 0x6008e5b to 0x6016a94, (0xdc39 bytes) 2024-04-23T21:21:34.306 INFO:teuthology.orchestra.run.smithi111.stdout:1114 write 0xa3f031b thru 0xa3f6b72 (0x6858 bytes) 2024-04-23T21:21:34.312 INFO:teuthology.orchestra.run.smithi111.stdout:1117 trunc from 0xc6631a9 to 0x87dad0b 2024-04-23T21:21:34.320 INFO:teuthology.orchestra.run.smithi111.stdout:1118 read 0x5269b6e thru 0x5275039 (0xb4cc bytes) 2024-04-23T21:21:34.321 INFO:teuthology.orchestra.run.smithi111.stdout:1119 write 0xde471a thru 0xdecbb8 (0x849f bytes) 2024-04-23T21:21:34.322 INFO:teuthology.orchestra.run.smithi111.stdout:1122 write 0x7eaf2ec thru 0x7eb29d3 (0x36e8 bytes) 2024-04-23T21:21:34.324 INFO:teuthology.orchestra.run.smithi111.stdout:1123 punch from 0x39f176d to 0x3a0042b, (0xecbe bytes) 2024-04-23T21:21:34.324 INFO:teuthology.orchestra.run.smithi111.stdout:1124 read 0x3240357 thru 0x3245473 (0x511d bytes) 2024-04-23T21:21:34.324 INFO:teuthology.orchestra.run.smithi111.stdout:1125 read 0x3429ff6 thru 0x342dc4b (0x3c56 bytes) 2024-04-23T21:21:34.325 INFO:teuthology.orchestra.run.smithi111.stdout:1127 write 0x7c17a0e thru 0x7c19333 (0x1926 bytes) 2024-04-23T21:21:34.326 INFO:teuthology.orchestra.run.smithi111.stdout:1129 write 0x5ef1917 thru 0x5eff32b (0xda15 bytes) 2024-04-23T21:21:34.333 INFO:teuthology.orchestra.run.smithi111.stdout:1130 writesame 0x8c94b97 thru 0x8ca0012 data_size 0x2(0xb47c bytes) 2024-04-23T21:21:34.335 INFO:teuthology.orchestra.run.smithi111.stdout:1131 read 0x42ea6af thru 0x42f1326 (0x6c78 bytes) 2024-04-23T21:21:34.335 INFO:teuthology.orchestra.run.smithi111.stdout:1132 punch from 0x40d5bf7 to 0x40d9c21, (0x402a bytes) 2024-04-23T21:21:34.335 INFO:teuthology.orchestra.run.smithi111.stdout:1133 read 0x3969207 thru 0x39787e4 (0xf5de bytes) 2024-04-23T21:21:34.336 INFO:teuthology.orchestra.run.smithi111.stdout:1136 write 0x1d00c23 thru 0x1d02b5f (0x1f3d bytes) 2024-04-23T21:21:34.344 INFO:teuthology.orchestra.run.smithi111.stdout:1137 write 0xa1d0e24 thru 0xa1dd717 (0xc8f4 bytes) 2024-04-23T21:21:34.349 INFO:teuthology.orchestra.run.smithi111.stdout:1138 trunc from 0xa1dd718 to 0xc8686fb 2024-04-23T21:21:34.354 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.355 INFO:teuthology.orchestra.run.smithi111.stdout:1140 write 0x669ae69 thru 0x66a1074 (0x620c bytes) 2024-04-23T21:21:34.356 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.362 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:34.362+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:34.365 INFO:teuthology.orchestra.run.smithi111.stdout:1143 trunc from 0xc8686fb to 0x4964045 2024-04-23T21:21:34.374 INFO:teuthology.orchestra.run.smithi111.stdout:1144 read 0x1fab953 thru 0x1fbb0b3 (0xf761 bytes) 2024-04-23T21:21:34.374 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.387 INFO:teuthology.orchestra.run.smithi111.stdout:1146 writesame 0xb3f80fc thru 0xb400e65 data_size 0x2(0x8d6a bytes) 2024-04-23T21:21:34.389 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.389 INFO:teuthology.orchestra.run.smithi111.stdout:1149 read 0xd8b2f7 thru 0xd9887e (0xd588 bytes) 2024-04-23T21:21:34.389 INFO:teuthology.orchestra.run.smithi111.stdout:1150 read 0x25d30cd thru 0x25d6c34 (0x3b68 bytes) 2024-04-23T21:21:34.389 INFO:teuthology.orchestra.run.smithi111.stdout:1151 writesame 0x8a7f797 thru 0x8a864ee data_size 0x4(0x6d58 bytes) 2024-04-23T21:21:34.393 INFO:teuthology.orchestra.run.smithi111.stdout:1152 read 0x2c234bd thru 0x2c26e66 (0x39aa bytes) 2024-04-23T21:21:34.393 INFO:teuthology.orchestra.run.smithi111.stdout:1153 read 0x7e3bcd1 thru 0x7e44350 (0x8680 bytes) 2024-04-23T21:21:34.393 INFO:teuthology.orchestra.run.smithi111.stdout:1154 writesame 0x71546b thru 0x717818 data_size 0x1(0x23ae bytes) 2024-04-23T21:21:34.397 INFO:teuthology.orchestra.run.smithi111.stdout:1155 writesame 0x92efa58 thru 0x92ff7ae data_size 0x7(0xfd57 bytes) 2024-04-23T21:21:34.403 INFO:teuthology.orchestra.run.smithi111.stdout:1156 trunc from 0xb400e66 to 0xcf360c7 2024-04-23T21:21:34.409 INFO:teuthology.orchestra.run.smithi111.stdout:1158 writesame 0x654f8fa thru 0x65517dd data_size 0x6(0x1ee4 bytes) 2024-04-23T21:21:34.410 INFO:teuthology.orchestra.run.smithi111.stdout:1160 write 0x2e7c226 thru 0x2e8315c (0x6f37 bytes) 2024-04-23T21:21:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:1163 punch from 0x3c6c41d to 0x3c7912b, (0xcd0e bytes) 2024-04-23T21:21:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:1164 read 0xc00df4c thru 0xc0182a4 (0xa359 bytes) 2024-04-23T21:21:34.412 INFO:teuthology.orchestra.run.smithi111.stdout:1165 write 0x52c9cae thru 0x52d8450 (0xe7a3 bytes) 2024-04-23T21:21:34.413 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:34.413 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:34.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.413 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:11041 3.15 3:a9218397:::rbd_data.111823663ad5.0000000000000007:head [stat out=16b,write 6069422~59299 in=59299b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.414499+0000", 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00045333, 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000472666, 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 11041 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414499+0000", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414499+0000", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414501+0000", 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7370000000000001e-06 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414525+0000", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4178000000000001e-05 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414537+0000", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1834000000000001e-05 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414543+0000", 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3910000000000003e-06 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414586+0000", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2929000000000002e-05 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414636+0000", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0043999999999997e-05 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 3", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.414713+0000", 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7411999999999997e-05 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.417 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.418 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.418 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:21:34.418 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:21:34.418 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1166 punch from 0x6787243 to 0x6795493, (0xe250 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1167 write 0x55d654f thru 0x55e1bea (0xb69c bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1168 writesame 0x28d90a1 thru 0x28d9645 data_size 0x5(0x5a5 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1169 punch from 0x17ccd62 to 0x17ceca7, (0x1f45 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1173 write 0xa037b55 thru 0xa037f89 (0x435 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1174 read 0x1a0e589 thru 0x1a1777e (0x91f6 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1176 write 0xaeeebec thru 0xaef10e3 (0x24f8 bytes) 2024-04-23T21:21:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:1179 write 0x30fc955 thru 0x31074dc (0xab88 bytes) 2024-04-23T21:21:34.421 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.421 INFO:teuthology.orchestra.run.smithi111.stdout:1182 read 0x9df947a thru 0x9e02682 (0x9209 bytes) 2024-04-23T21:21:34.421 INFO:teuthology.orchestra.run.smithi111.stdout:1185 read 0x249d4ce thru 0x24a81d1 (0xad04 bytes) 2024-04-23T21:21:34.421 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.422 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.422 INFO:teuthology.orchestra.run.smithi111.stdout:1188 read 0xa28ccb3 thru 0xa28ffb8 (0x3306 bytes) 2024-04-23T21:21:34.422 INFO:teuthology.orchestra.run.smithi111.stdout:1190 read 0x2aee532 thru 0x2af0a0a (0x24d9 bytes) 2024-04-23T21:21:34.422 INFO:teuthology.orchestra.run.smithi111.stdout:1191 write 0xb101834 thru 0xb101add (0x2aa bytes) 2024-04-23T21:21:34.423 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:34.424 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.424 INFO:teuthology.orchestra.run.smithi111.stdout:1194 writesame 0x9e6d720 thru 0x9e74101 data_size 0x1(0x69e2 bytes) 2024-04-23T21:21:34.438 INFO:teuthology.orchestra.run.smithi111.stdout:1195 trunc from 0xcf360c7 to 0xc34716c 2024-04-23T21:21:34.445 INFO:teuthology.orchestra.run.smithi111.stdout:1196 write 0x6210247 thru 0x621fe49 (0xfc03 bytes) 2024-04-23T21:21:34.450 INFO:teuthology.orchestra.run.smithi111.stdout:1197 trunc from 0xc34716c to 0xed12313 2024-04-23T21:21:34.455 INFO:teuthology.orchestra.run.smithi111.stdout:1198 read 0x172548b thru 0x172a560 (0x50d6 bytes) 2024-04-23T21:21:34.455 INFO:teuthology.orchestra.run.smithi111.stdout:1199 writesame 0xd7c6cfa thru 0xd7c962f data_size 0x19(0x2936 bytes) 2024-04-23T21:21:34.457 INFO:teuthology.orchestra.run.smithi111.stdout:1200 write 0xae29ad2 thru 0xae2bebe (0x23ed bytes) 2024-04-23T21:21:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:34.459 INFO:teuthology.orchestra.run.smithi111.stdout:1206 read 0x515cfd7 thru 0x515f231 (0x225b bytes) 2024-04-23T21:21:34.459 INFO:teuthology.orchestra.run.smithi111.stdout:1207 punch from 0x3e7999e to 0x3e82551, (0x8bb3 bytes) 2024-04-23T21:21:34.459 INFO:teuthology.orchestra.run.smithi111.stdout:1208 clone 9 order 20 su 262144 sc 6 2024-04-23T21:21:34.463 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:34.464+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:34.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:34.527 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:34.527 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:34.527 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:34.527 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:34.527 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:34.535 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:34.579 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:34.580+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3197 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2916352~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218565692, 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024247707, 2024-04-23T21:21:34.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3197 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420534+0000", 2024-04-23T21:21:34.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.074e-06 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420544+0000", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3370000000000006e-06 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420549+0000", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9030000000000003e-06 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444634+0000", 2024-04-23T21:21:34.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024085517000000001 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444713+0000", 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444780+0000", 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6797000000000004e-05 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3198 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2981888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218542083999999, 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024356404000000002, 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3198 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420556+0000", 2024-04-23T21:21:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420558+0000", 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420564+0000", 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7089999999999999e-06 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420568+0000", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6280000000000004e-06 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444802+0000", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024234169999999999 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444844+0000", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444912+0000", 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8367999999999993e-05 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218523131, 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:21:34.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:21:34.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:21:34.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218495768, 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:21:34.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218481519000001, 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:34.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:21:34.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218459276000001, 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.21839379, 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:21:34.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:21:34.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218368048, 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.673 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:21:34.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:21:34.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:21:34.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218344039, 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.679 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:34.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.21832148, 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:21:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:21:34.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:21:34.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218301676999999, 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.689 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:21:34.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:21:34.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.695 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218282191, 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.696 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.703 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218262392, 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:21:34.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:34.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:21:34.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.21824269, 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:21:34.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:34.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218186564, 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:21:34.716 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:21:34.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:21:34.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218149343, 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:21:34.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:21:34.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:21:34.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218099029999999, 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.731 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:21:34.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:21:34.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:34.738 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218066135000001, 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218043830999999, 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:21:34.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.218023414999999, 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:34.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:34.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:21:34.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:21:34.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:34.752 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:34.754 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:34.754 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:34.754 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:34.755 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:34.755 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:34.756 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:34.756 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:34.756 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:34.757 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:34.757 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:34.757 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:34.758 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:34.758 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:34.758 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:34.758 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:34.759 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:34.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:34.760 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:34.754+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:34.855 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:34.856+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:34.857 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:21:34.913 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:34.913 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:34.918 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone8 from 0xed12313 (overlap 0x4964045) to 0x24b2022 2024-04-23T21:21:34.923 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:34.956 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:34.957+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:35.002 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-04-23T21:21:35.013 INFO:teuthology.orchestra.run.smithi111.stdout:1210 trunc from 0xed12313 to 0x6886b0d 2024-04-23T21:21:35.029 INFO:teuthology.orchestra.run.smithi111.stdout:1211 punch from 0x10ceebd to 0x10d02e5, (0x1428 bytes) 2024-04-23T21:21:35.033 INFO:teuthology.orchestra.run.smithi111.stdout:1214 writesame 0x6dce56f thru 0x6dd9606 data_size 0x4(0xb098 bytes) 2024-04-23T21:21:35.034 INFO:teuthology.orchestra.run.smithi111.stdout:1215 write 0x13d9b30 thru 0x13da247 (0x718 bytes) 2024-04-23T21:21:35.037 INFO:teuthology.orchestra.run.smithi111.stdout:1216 writesame 0x3716021 thru 0x3722c49 data_size 0x1(0xcc29 bytes) 2024-04-23T21:21:35.040 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.041 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:35.056 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:35.057+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:35.062 INFO:teuthology.orchestra.run.smithi111.stdout:1217 flatten 2024-04-23T21:21:35.108 INFO:teuthology.orchestra.run.smithi111.stdout:1218 write 0x280eb36 thru 0x281d445 (0xe910 bytes) 2024-04-23T21:21:35.118 INFO:teuthology.orchestra.run.smithi111.stdout:1219 write 0x7e3a257 thru 0x7e42909 (0x86b3 bytes) 2024-04-23T21:21:35.121 INFO:teuthology.orchestra.run.smithi111.stdout:1220 read 0x64d71c1 thru 0x64dfece (0x8d0e bytes) 2024-04-23T21:21:35.121 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.121 INFO:teuthology.orchestra.run.smithi111.stdout:1224 writesame 0x28cd55d thru 0x28dd3eb data_size 0x1(0xfe8f bytes) 2024-04-23T21:21:35.151 INFO:teuthology.orchestra.run.smithi111.stdout:1225 read 0x7bd4580 thru 0x7be1125 (0xcba6 bytes) 2024-04-23T21:21:35.151 INFO:teuthology.orchestra.run.smithi111.stdout:1226 read 0x1034571 thru 0x1042e4c (0xe8dc bytes) 2024-04-23T21:21:35.157 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:35.157+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:35.163 INFO:teuthology.orchestra.run.smithi111.stdout:1227 write 0xc971b89 thru 0xc98050a (0xe982 bytes) 2024-04-23T21:21:35.170 INFO:teuthology.orchestra.run.smithi111.stdout:1228 trunc from 0xc98050b to 0x9fe48f2 2024-04-23T21:21:35.172 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.172 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.180 INFO:teuthology.orchestra.run.smithi111.stdout:1233 write 0x50456f9 thru 0x5053e14 (0xe71c bytes) 2024-04-23T21:21:35.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:35.188 INFO:teuthology.orchestra.run.smithi111.stdout:1234 write 0xa7acf47 thru 0xa7adac2 (0xb7c bytes) 2024-04-23T21:21:35.191 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.191 INFO:teuthology.orchestra.run.smithi111.stdout:1236 write 0x8e2a5a7 thru 0x8e38736 (0xe190 bytes) 2024-04-23T21:21:35.204 INFO:teuthology.orchestra.run.smithi111.stdout:1237 write 0xe988f37 thru 0xe993d66 (0xae30 bytes) 2024-04-23T21:21:35.212 INFO:teuthology.orchestra.run.smithi111.stdout:1238 trunc from 0xe993d67 to 0xb12419a 2024-04-23T21:21:35.221 INFO:teuthology.orchestra.run.smithi111.stdout:1239 read 0xa22f10e thru 0xa23cd84 (0xdc77 bytes) 2024-04-23T21:21:35.222 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.222 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.230 INFO:teuthology.orchestra.run.smithi111.stdout:1242 write 0xe460b0f thru 0xe46508b (0x457d bytes) 2024-04-23T21:21:35.233 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.233 INFO:teuthology.orchestra.run.smithi111.stdout:1244 read 0xb25cfa0 thru 0xb26a8b9 (0xd91a bytes) 2024-04-23T21:21:35.240 INFO:teuthology.orchestra.run.smithi111.stdout:1246 trunc from 0xe46508c to 0x879900f 2024-04-23T21:21:35.257 INFO:teuthology.orchestra.run.smithi111.stdout:1247 trunc from 0x879900f to 0x343478f 2024-04-23T21:21:35.258 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:35.258+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:35.271 INFO:teuthology.orchestra.run.smithi111.stdout:1248 read 0x11cbcfd thru 0x11d2300 (0x6604 bytes) 2024-04-23T21:21:35.271 INFO:teuthology.orchestra.run.smithi111.stdout:1249 punch from 0x201d6d8 to 0x2025f09, (0x8831 bytes) 2024-04-23T21:21:35.271 INFO:teuthology.orchestra.run.smithi111.stdout:1250 read 0x167d5a1 thru 0x1686420 (0x8e80 bytes) 2024-04-23T21:21:35.288 INFO:teuthology.orchestra.run.smithi111.stdout:1251 writesame 0xd6c92ee thru 0xd6c97d6 data_size 0x1(0x4e9 bytes) 2024-04-23T21:21:35.288 INFO:teuthology.orchestra.run.smithi111.stdout:1255 read 0xd569957 thru 0xd56c629 (0x2cd3 bytes) 2024-04-23T21:21:35.288 INFO:teuthology.orchestra.run.smithi111.stdout:1256 write 0x2b4ff22 thru 0x2b56ff7 (0x70d6 bytes) 2024-04-23T21:21:35.289 INFO:teuthology.orchestra.run.smithi111.stdout:1257 read 0x6d758cb thru 0x6d7d7e8 (0x7f1e bytes) 2024-04-23T21:21:35.289 INFO:teuthology.orchestra.run.smithi111.stdout:1260 write 0x52400d thru 0x52d619 (0x960d bytes) 2024-04-23T21:21:35.291 INFO:teuthology.orchestra.run.smithi111.stdout:1261 read 0x1d84bce thru 0x1d8a44f (0x5882 bytes) 2024-04-23T21:21:35.304 INFO:teuthology.orchestra.run.smithi111.stdout:1262 trunc from 0xd6c97d7 to 0x2360168 2024-04-23T21:21:35.307 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.307 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.316 INFO:teuthology.orchestra.run.smithi111.stdout:1263 write 0x1ebdf0a thru 0x1ec2103 (0x41fa bytes) 2024-04-23T21:21:35.317 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:35.320 INFO:teuthology.orchestra.run.smithi111.stdout:1264 read 0x9a712 thru 0x9d737 (0x3026 bytes) 2024-04-23T21:21:35.320 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.320 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.329 INFO:teuthology.orchestra.run.smithi111.stdout:1268 trunc from 0x2360168 to 0xa1a60c7 2024-04-23T21:21:35.335 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.335 INFO:teuthology.orchestra.run.smithi111.stdout:1271 read 0x8f2370f thru 0x8f2fd28 (0xc61a bytes) 2024-04-23T21:21:35.335 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.335 INFO:teuthology.orchestra.run.smithi111.stdout:1273 writesame 0x624c78 thru 0x63090f data_size 0x8e(0xbc98 bytes) 2024-04-23T21:21:35.345 INFO:teuthology.orchestra.run.smithi111.stdout:1274 write 0xcbc74be thru 0xcbd3a55 (0xc598 bytes) 2024-04-23T21:21:35.348 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.348 INFO:teuthology.orchestra.run.smithi111.stdout:1277 punch from 0xc54c142 to 0xc54ce03, (0xcc1 bytes) 2024-04-23T21:21:35.358 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:35.358+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:35.363 INFO:teuthology.orchestra.run.smithi111.stdout:1278 trunc from 0xcbd3a56 to 0x7e6442 2024-04-23T21:21:35.391 INFO:teuthology.orchestra.run.smithi111.stdout:1279 write 0x794e0b6 thru 0x795850d (0xa458 bytes) 2024-04-23T21:21:35.395 INFO:teuthology.orchestra.run.smithi111.stdout:1280 write 0x7185284 thru 0x71940b1 (0xee2e bytes) 2024-04-23T21:21:35.405 INFO:teuthology.orchestra.run.smithi111.stdout:1281 trunc from 0x795850e to 0xc154ea6 2024-04-23T21:21:35.410 INFO:teuthology.orchestra.run.smithi111.stdout:1282 write 0x454b1df thru 0x455a4c3 (0xf2e5 bytes) 2024-04-23T21:21:35.412 INFO:teuthology.orchestra.run.smithi111.stdout:1283 writesame 0x3d7e06f thru 0x3d880dc data_size 0x6f(0xa06e bytes) 2024-04-23T21:21:35.419 INFO:teuthology.orchestra.run.smithi111.stdout:1284 read 0x11df7ab thru 0x11ef4af (0xfd05 bytes) 2024-04-23T21:21:35.419 INFO:teuthology.orchestra.run.smithi111.stdout:1285 punch from 0xafaca22 to 0xafb5844, (0x8e22 bytes) 2024-04-23T21:21:35.419 INFO:teuthology.orchestra.run.smithi111.stdout:1287 writesame 0x56243a9 thru 0x56271be data_size 0x11(0x2e16 bytes) 2024-04-23T21:21:35.422 INFO:teuthology.orchestra.run.smithi111.stdout:1288 read 0x729e630 thru 0x72ab050 (0xca21 bytes) 2024-04-23T21:21:35.422 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.422 INFO:teuthology.orchestra.run.smithi111.stdout:1290 read 0x37b28f4 thru 0x37b9dac (0x74b9 bytes) 2024-04-23T21:21:35.422 INFO:teuthology.orchestra.run.smithi111.stdout:1291 read 0x7840949 thru 0x7849602 (0x8cba bytes) 2024-04-23T21:21:35.425 INFO:teuthology.orchestra.run.smithi111.stdout:1292 trunc from 0xc154ea6 to 0xec131a4 2024-04-23T21:21:35.430 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.430 INFO:teuthology.orchestra.run.smithi111.stdout:1294 read 0x753569a thru 0x7541ffa (0xc961 bytes) 2024-04-23T21:21:35.430 INFO:teuthology.orchestra.run.smithi111.stdout:1298 write 0x19ef5db thru 0x19fce55 (0xd87b bytes) 2024-04-23T21:21:35.433 INFO:teuthology.orchestra.run.smithi111.stdout:1299 read 0x8e68b53 thru 0x8e731ff (0xa6ad bytes) 2024-04-23T21:21:35.433 INFO:teuthology.orchestra.run.smithi111.stdout:1301 write 0x8ae50e thru 0x8ae8de (0x3d1 bytes) 2024-04-23T21:21:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:1303 read 0x169a12a thru 0x16a6f1f (0xcdf6 bytes) 2024-04-23T21:21:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:1304 punch from 0xa030bc9 to 0xa0364c8, (0x58ff bytes) 2024-04-23T21:21:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:1305 punch from 0xde87ea9 to 0xde930d5, (0xb22c bytes) 2024-04-23T21:21:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:1307 punch from 0x816c5ea to 0x81740fb, (0x7b11 bytes) 2024-04-23T21:21:35.435 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.439 INFO:teuthology.orchestra.run.smithi111.stdout:1309 write 0xec5caf3 thru 0xec6319e (0x66ac bytes) 2024-04-23T21:21:35.439 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.439 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.443 INFO:teuthology.orchestra.run.smithi111.stdout:1310 trunc from 0xec6319f to 0xd349fbe 2024-04-23T21:21:35.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:35.455 INFO:teuthology.orchestra.run.smithi111.stdout:1312 read 0xc8ed91e thru 0xc8fabb6 (0xd299 bytes) 2024-04-23T21:21:35.456 INFO:teuthology.orchestra.run.smithi111.stdout:1313 trunc from 0xd349fbe to 0xb3f1e49 2024-04-23T21:21:35.458 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:35.459+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:35.461 INFO:teuthology.orchestra.run.smithi111.stdout:1314 read 0x811a8df thru 0x8122de5 (0x8507 bytes) 2024-04-23T21:21:35.470 INFO:teuthology.orchestra.run.smithi111.stdout:1315 writesame 0xe678434 thru 0xe67d4f9 data_size 0xe(0x50c6 bytes) 2024-04-23T21:21:35.471 INFO:teuthology.orchestra.run.smithi111.stdout:1317 write 0x690e59b thru 0x6915212 (0x6c78 bytes) 2024-04-23T21:21:35.474 INFO:teuthology.orchestra.run.smithi111.stdout:1319 punch from 0x98499ba to 0x9851d36, (0x837c bytes) 2024-04-23T21:21:35.474 INFO:teuthology.orchestra.run.smithi111.stdout:1320 read 0x71e893 thru 0x7279ee (0x915c bytes) 2024-04-23T21:21:35.474 INFO:teuthology.orchestra.run.smithi111.stdout:1321 write 0x752996a thru 0x752c7ce (0x2e65 bytes) 2024-04-23T21:21:35.478 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:1323 read 0x60741a7 thru 0x607772a (0x3584 bytes) 2024-04-23T21:21:35.479 INFO:teuthology.orchestra.run.smithi111.stdout:1324 write 0x23ea7c0 thru 0x23f36de (0x8f1f bytes) 2024-04-23T21:21:35.499 INFO:teuthology.orchestra.run.smithi111.stdout:1326 trunc from 0xe67d4fa to 0x2faae77 2024-04-23T21:21:35.510 INFO:teuthology.orchestra.run.smithi111.stdout:1328 read 0x2c318cf thru 0x2c36831 (0x4f63 bytes) 2024-04-23T21:21:35.511 INFO:teuthology.orchestra.run.smithi111.stdout:1329 read 0x1323f3 thru 0x13ed68 (0xc976 bytes) 2024-04-23T21:21:35.525 INFO:teuthology.orchestra.run.smithi111.stdout:1331 write 0xb141075 thru 0xb146c28 (0x5bb4 bytes) 2024-04-23T21:21:35.528 INFO:teuthology.orchestra.run.smithi111.stdout:1332 writesame 0x85e3134 thru 0x85e42f6 data_size 0x1(0x11c3 bytes) 2024-04-23T21:21:35.532 INFO:teuthology.orchestra.run.smithi111.stdout:1333 write 0x27daa21 thru 0x27ea3a8 (0xf988 bytes) 2024-04-23T21:21:35.545 INFO:teuthology.orchestra.run.smithi111.stdout:1334 trunc from 0xb146c29 to 0x28c4e45 2024-04-23T21:21:35.559 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:35.560+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:35.573 INFO:teuthology.orchestra.run.smithi111.stdout:1335 writesame 0xdbebf65 thru 0xdbf208d data_size 0x1(0x6129 bytes) 2024-04-23T21:21:35.574 INFO:teuthology.orchestra.run.smithi111.stdout:1336 writesame 0x28cbc92 thru 0x28d9f0b data_size 0x9(0xe27a bytes) 2024-04-23T21:21:35.578 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.578 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.579 INFO:teuthology.orchestra.run.smithi111.stdout:1337 writesame 0xe471fc thru 0xe4d5c3 data_size 0x1f(0x63c8 bytes) 2024-04-23T21:21:35.589 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:35.589 INFO:teuthology.orchestra.run.smithi111.stdout:1339 write 0xe4add22 thru 0xe4b2d31 (0x5010 bytes) 2024-04-23T21:21:35.592 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:35.592 INFO:teuthology.orchestra.run.smithi111.stdout:1342 clone 10 order 25 su 8388608 sc 13 2024-04-23T21:21:35.660 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:35.662+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:35.694 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.695 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.703 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:35.761 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:35.761+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:35.804 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.805 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:35.863 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:35.864+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:35.962 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:35.962 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:35.963 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:35.965+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:35.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:36.067 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:36.067+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:36.074 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-04-23T21:21:36.078 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.078 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.087 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:36.119 INFO:teuthology.orchestra.run.smithi111.stdout:1344 write 0xcf7132d thru 0xcf77de7 (0x6abb bytes) 2024-04-23T21:21:36.126 INFO:teuthology.orchestra.run.smithi111.stdout:1345 read 0x327d0d9 thru 0x32893f8 (0xc320 bytes) 2024-04-23T21:21:36.127 INFO:teuthology.orchestra.run.smithi111.stdout:1346 writesame 0xdf49b19 thru 0xdf5258f data_size 0x1(0x8a77 bytes) 2024-04-23T21:21:36.146 INFO:teuthology.orchestra.run.smithi111.stdout:1347 punch from 0xa8bea14 to 0xa8cc939, (0xdf25 bytes) 2024-04-23T21:21:36.146 INFO:teuthology.orchestra.run.smithi111.stdout:1350 write 0xd6b1a44 thru 0xd6bb37c (0x9939 bytes) 2024-04-23T21:21:36.160 INFO:teuthology.orchestra.run.smithi111.stdout:1352 read 0x83c70ad thru 0x83c95fa (0x254e bytes) 2024-04-23T21:21:36.160 INFO:teuthology.orchestra.run.smithi111.stdout:1353 read 0xdc91020 thru 0xdc96adb (0x5abc bytes) 2024-04-23T21:21:36.160 INFO:teuthology.orchestra.run.smithi111.stdout:1354 write 0x54d11f5 thru 0x54d9c3a (0x8a46 bytes) 2024-04-23T21:21:36.164 INFO:teuthology.orchestra.run.smithi111.stdout:1355 writesame 0x6449387 thru 0x644a95d data_size 0x1(0x15d7 bytes) 2024-04-23T21:21:36.167 INFO:teuthology.orchestra.run.smithi111.stdout:1356 writesame 0xdae7bbc thru 0xdaf3598 data_size 0x1(0xb9dd bytes) 2024-04-23T21:21:36.167 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:36.167+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:36.187 INFO:teuthology.orchestra.run.smithi111.stdout:1357 write 0x202241 thru 0x203107 (0xec7 bytes) 2024-04-23T21:21:36.189 INFO:teuthology.orchestra.run.smithi111.stdout:1358 writesame 0x64a6a06 thru 0x64a8567 data_size 0x5(0x1b62 bytes) 2024-04-23T21:21:36.195 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.195 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.196 INFO:teuthology.orchestra.run.smithi111.stdout:1359 trunc from 0xe4b2d32 to 0xa10e1e9 2024-04-23T21:21:36.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:36.209 INFO:teuthology.orchestra.run.smithi111.stdout:1360 read 0xd9418b thru 0xd954a8 (0x131e bytes) 2024-04-23T21:21:36.209 INFO:teuthology.orchestra.run.smithi111.stdout:1361 read 0x3c0ae70 thru 0x3c1a8ee (0xfa7f bytes) 2024-04-23T21:21:36.210 INFO:teuthology.orchestra.run.smithi111.stdout:1362 writesame 0x80034e3 thru 0x800d65f data_size 0x1(0xa17d bytes) 2024-04-23T21:21:36.232 INFO:teuthology.orchestra.run.smithi111.stdout:1363 write 0x773dcf2 thru 0x774739a (0x96a9 bytes) 2024-04-23T21:21:36.234 INFO:teuthology.orchestra.run.smithi111.stdout:1365 punch from 0x9e057e1 to 0x9e0aef3, (0x5712 bytes) 2024-04-23T21:21:36.234 INFO:teuthology.orchestra.run.smithi111.stdout:1367 read 0x94a8df9 thru 0x94af795 (0x699d bytes) 2024-04-23T21:21:36.245 INFO:teuthology.orchestra.run.smithi111.stdout:1369 write 0xeca993b thru 0xecb3cb1 (0xa377 bytes) 2024-04-23T21:21:36.247 INFO:teuthology.orchestra.run.smithi111.stdout:1370 read 0x733eb15 thru 0x73462c7 (0x77b3 bytes) 2024-04-23T21:21:36.254 INFO:teuthology.orchestra.run.smithi111.stdout:1371 trunc from 0xecb3cb2 to 0x967cc2b 2024-04-23T21:21:36.264 INFO:teuthology.orchestra.run.smithi111.stdout:1372 punch from 0x686ab0e to 0x6874918, (0x9e0a bytes) 2024-04-23T21:21:36.265 INFO:teuthology.orchestra.run.smithi111.stdout:1373 write 0x5b4abec thru 0x5b58c05 (0xe01a bytes) 2024-04-23T21:21:36.267 INFO:teuthology.orchestra.run.smithi111.stdout:1374 punch from 0x861f87 to 0x864c6f, (0x2ce8 bytes) 2024-04-23T21:21:36.267 INFO:teuthology.orchestra.run.smithi111.stdout:1375 punch from 0x3edd400 to 0x3ee3b0c, (0x670c bytes) 2024-04-23T21:21:36.267 INFO:teuthology.orchestra.run.smithi111.stdout:1376 writesame 0x16bb797 thru 0x16c4ff7 data_size 0x1(0x9861 bytes) 2024-04-23T21:21:36.268 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:36.268+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:36.289 INFO:teuthology.orchestra.run.smithi111.stdout:1377 read 0x636a2b thru 0x638df6 (0x23cc bytes) 2024-04-23T21:21:36.296 INFO:teuthology.orchestra.run.smithi111.stdout:1378 write 0x989093c thru 0x989d5a4 (0xcc69 bytes) 2024-04-23T21:21:36.307 INFO:teuthology.orchestra.run.smithi111.stdout:1379 trunc from 0x989d5a5 to 0x2128fca 2024-04-23T21:21:36.321 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.321 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:36.339 INFO:teuthology.orchestra.run.smithi111.stdout:1380 write 0x3560c0c thru 0x356b3d6 (0xa7cb bytes) 2024-04-23T21:21:36.343 INFO:teuthology.orchestra.run.smithi111.stdout:1382 read 0x563fa7 thru 0x5671a0 (0x31fa bytes) 2024-04-23T21:21:36.344 INFO:teuthology.orchestra.run.smithi111.stdout:1383 write 0xbe062 thru 0xbe52f (0x4ce bytes) 2024-04-23T21:21:36.345 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:36.354 INFO:teuthology.orchestra.run.smithi111.stdout:1385 write 0x67a52aa thru 0x67a635d (0x10b4 bytes) 2024-04-23T21:21:36.357 INFO:teuthology.orchestra.run.smithi111.stdout:1387 write 0x21c445e thru 0x21d3335 (0xeed8 bytes) 2024-04-23T21:21:36.359 INFO:teuthology.orchestra.run.smithi111.stdout:1389 read 0x439454b thru 0x439e419 (0x9ecf bytes) 2024-04-23T21:21:36.372 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:36.373+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:36.374 INFO:teuthology.orchestra.run.smithi111.stdout:1391 writesame 0xd815213 thru 0xd81c53d data_size 0x1(0x732b bytes) 2024-04-23T21:21:36.375 INFO:teuthology.orchestra.run.smithi111.stdout:1392 write 0x48eebf4 thru 0x48f5abd (0x6eca bytes) 2024-04-23T21:21:36.379 INFO:teuthology.orchestra.run.smithi111.stdout:1393 trunc from 0xd81c53e to 0xc49bc93 2024-04-23T21:21:36.387 INFO:teuthology.orchestra.run.smithi111.stdout:1394 punch from 0x683f1e3 to 0x684ebd9, (0xf9f6 bytes) 2024-04-23T21:21:36.387 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:36.387 INFO:teuthology.orchestra.run.smithi111.stdout:1396 read 0x15cb291 thru 0x15cd0b5 (0x1e25 bytes) 2024-04-23T21:21:36.396 INFO:teuthology.orchestra.run.smithi111.stdout:1398 trunc from 0xc49bc93 to 0x4730b33 2024-04-23T21:21:36.413 INFO:teuthology.orchestra.run.smithi111.stdout:1400 read 0x4634c12 thru 0x463dfe5 (0x93d4 bytes) 2024-04-23T21:21:36.413 INFO:teuthology.orchestra.run.smithi111.stdout:1401 punch from 0x10545aa to 0x10633ae, (0xee04 bytes) 2024-04-23T21:21:36.413 INFO:teuthology.orchestra.run.smithi111.stdout:1402 read 0x471e210 thru 0x4727973 (0x9764 bytes) 2024-04-23T21:21:36.413 INFO:teuthology.orchestra.run.smithi111.stdout:1403 punch from 0x46e1c8e to 0x46e5de8, (0x415a bytes) 2024-04-23T21:21:36.413 INFO:teuthology.orchestra.run.smithi111.stdout:1404 punch from 0x26f8ad5 to 0x2702186, (0x96b1 bytes) 2024-04-23T21:21:36.428 INFO:teuthology.orchestra.run.smithi111.stdout:1405 write 0xc8b67bb thru 0xc8c6711 (0xff57 bytes) 2024-04-23T21:21:36.431 INFO:teuthology.orchestra.run.smithi111.stdout:1407 write 0x897ecb0 thru 0x898921f (0xa570 bytes) 2024-04-23T21:21:36.444 INFO:teuthology.orchestra.run.smithi111.stdout:1409 trunc from 0xc8c6712 to 0x28f3873 2024-04-23T21:21:36.457 INFO:teuthology.orchestra.run.smithi111.stdout:1410 clone 11 order 23 su 1048576 sc 15 2024-04-23T21:21:36.470 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:36.470+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:36.480 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.481 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:36.571 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:36.572+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:36.592 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.592 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:36.685 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:36.686+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:36.744 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.744 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:36.760 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:36.793 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:36.794+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:36.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:36.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:21:36.856 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-23T21:21:36.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-04-23T21:21:36.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:21:36.891 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:36.891 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:36.892 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:36.891+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:36.892 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:36.899 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:36.899 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:36.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:36.924 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone10 from 0x28f3873 (overlap 0x2128fca) to 0x0 2024-04-23T21:21:36.926 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:36.926 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:36.963 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:36.975 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:36.975+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:36.978 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-04-23T21:21:36.995 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:36.995 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:37.015 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:37.015 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:37.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:37.027 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:37.028 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:37.036 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-04-23T21:21:37.052 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:37.052 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:37.070 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:37.070 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:37.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:37.075 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:37.076+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:37.131 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:37.131 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:37.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:2565 3.1e e23/18 3:7adf3bd0:::rbd_data.11183c92f5c.0000000000000005:head v 23'23, mlcod=22'21)", 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.760024834999999, 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063423730000000001, 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.74e-07 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374893+0000", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038888899999999999 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374909+0000", 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374921+0000", 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2031000000000001e-05 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374987+0000", 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5827000000000005e-05 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.375003+0000", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6393e-05 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377299+0000", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002295503 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377346+0000", 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7475000000000002e-05 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7401 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,truncate 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.6856666279999999, 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036060879999999999, 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7401 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445362+0000", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445363+0000", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7000000000000003e-07 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445969+0000", 2024-04-23T21:21:37.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00060587 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445981+0000", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1731e-05 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.445986+0000", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446040+0000", 2024-04-23T21:21:37.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3427000000000002e-05 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446176+0000", 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013631200000000001 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.446400+0000", 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022467900000000001 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.447547+0000", 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001146527 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448901+0000", 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013539859999999999 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448939+0000", 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7608000000000003e-05 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.448968+0000", 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9410000000000001e-05 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7476 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [stat,write 3276800~32768] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.5164040849999996, 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0039134579999999999, 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.141 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7476 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614624+0000", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614627+0000", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4770000000000002e-06 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614667+0000", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9662000000000001e-05 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614687+0000", 2024-04-23T21:21:37.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0811000000000001e-05 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614698+0000", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0465000000000001e-05 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614755+0000", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6702e-05 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614809+0000", 2024-04-23T21:21:37.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4886000000000003e-05 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.614900+0000", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0829999999999996e-05 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.615705+0000", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00080430599999999997 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618432+0000", 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027271420000000001 2024-04-23T21:21:37.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618490+0000", 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8304000000000003e-05 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.618538+0000", 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7873000000000001e-05 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:7498 3.10 3:0c3dd7f6:::rbd_data.111830ac55a1.0000000000000000:head [stat,truncate 3768320] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 7.4908254330000004, 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0050186079999999999, 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 7498 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640203+0000", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640204+0000", 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1319999999999999e-06 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640209+0000", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0420000000000002e-06 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640223+0000", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3813e-05 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640232+0000", 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.473e-06 2024-04-23T21:21:37.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640292+0000", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0401e-05 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640374+0000", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1729000000000003e-05 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.640503+0000", 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000129348 2024-04-23T21:21:37.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.641060+0000", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00055648700000000004 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645148+0000", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004088688 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645184+0000", 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5491000000000003e-05 2024-04-23T21:21:37.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:29.645222+0000", 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8003999999999997e-05 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:8046 3.a 3:56ba5b24:::rbd_data.111830ac55a1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.8880830299999998, 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014462938, 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.152 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 8046 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:37.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242952+0000", 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4650000000000002e-06 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242966+0000", 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3708e-05 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242974+0000", 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0369999999999995e-06 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243016+0000", 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2769000000000003e-05 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243063+0000", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999997e-05 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.245737+0000", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026734939999999998 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257269+0000", 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011532351 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257289+0000", 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0605e-05 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257352+0000", 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2535999999999999e-05 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257408+0000", 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6450000000000003e-05 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:8046 3.a e27/18 ondisk, result = 0)", 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 6.8777295760000001, 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003980267, 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.158 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253299+0000", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253300+0000", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5099999999999998e-07 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253303+0000", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5279999999999999e-06 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253315+0000", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1365999999999999e-05 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.253324+0000", 2024-04-23T21:21:37.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9949999999999994e-06 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257249+0000", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003925057 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257261+0000", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2340999999999999e-05 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257279+0000", 2024-04-23T21:21:37.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8128999999999999e-05 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10823 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238309+0000", 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.892719628, 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002913166, 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10823 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238309+0000", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238309+0000", 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238310+0000", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.153e-06 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238318+0000", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2779999999999993e-06 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238334+0000", 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6113999999999999e-05 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238342+0000", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9559999999999997e-06 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238447+0000", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104962 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238548+0000", 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100704 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238901+0000", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00035287699999999998 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241128+0000", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022273779999999999 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241171+0000", 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2969999999999997e-05 2024-04-23T21:21:37.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241197+0000", 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5261000000000001e-05 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241222+0000", 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5513000000000001e-05 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.167 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10825 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8925721009999998, 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0062412700000000002, 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10825 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:37.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238457+0000", 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3099999999999996e-07 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238463+0000", 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5090000000000001e-06 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238474+0000", 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1455e-05 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238484+0000", 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5480000000000007e-06 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240357+0000", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001872964 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240446+0000", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9569999999999998e-05 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240766+0000", 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031934500000000001 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244476+0000", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037105229999999999 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244570+0000", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3869999999999994e-05 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244641+0000", 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0737999999999999e-05 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244698+0000", 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6817000000000002e-05 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10826 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.892447448, 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080318590000000006, 2024-04-23T21:21:37.173 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10826 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238582+0000", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0399999999999995e-07 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238586+0000", 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2740000000000001e-06 2024-04-23T21:21:37.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238595+0000", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1220000000000001e-06 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238602+0000", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3559999999999998e-06 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241310+0000", 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027081649999999998 2024-04-23T21:21:37.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241371+0000", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0829000000000003e-05 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241647+0000", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027606199999999999 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244483+0000", 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028360680000000002 2024-04-23T21:21:37.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246556+0000", 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002073713 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246588+0000", 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1829000000000003e-05 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246613+0000", 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4737e-05 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10827 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8924094990000002, 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008083504, 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10827 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238620+0000", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-07 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238622+0000", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6479999999999999e-06 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238628+0000", 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3489999999999999e-06 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238631+0000", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2990000000000001e-06 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242043+0000", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034116889999999999 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242092+0000", 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9542000000000001e-05 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242319+0000", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022715600000000001 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245447+0000", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031279509999999999 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246652+0000", 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001204526 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246681+0000", 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8892999999999999e-05 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246703+0000", 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1869999999999999e-05 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.184 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10828 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8923839249999999, 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011309565000000001, 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10828 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-07 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238648+0000", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8619999999999999e-06 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238656+0000", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0909999999999993e-06 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238661+0000", 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2349999999999999e-06 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242733+0000", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040721500000000001 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242819+0000", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5457999999999994e-05 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243185+0000", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036640300000000001 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246517+0000", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003332149 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249839+0000", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033218850000000001 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249896+0000", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7085e-05 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249954+0000", 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7782999999999999e-05 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10829 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8923573579999999, 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011463991999999999, 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10829 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238672+0000", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8599999999999998e-07 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238674+0000", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6639999999999998e-06 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238681+0000", 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1639999999999998e-06 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238689+0000", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0800000000000006e-06 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243618+0000", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049292119999999997 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243705+0000", 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7101000000000005e-05 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244084+0000", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037872300000000003 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246524+0000", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002440331 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250022+0000", 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034982429999999998 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250081+0000", 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8403000000000003e-05 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250135+0000", 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4585e-05 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10830 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.195 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8922524040000002, 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011515780999999999, 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10830 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238777+0000", 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0200000000000001e-07 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238781+0000", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238790+0000", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1169999999999993e-06 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238797+0000", 2024-04-23T21:21:37.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8419999999999999e-06 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244765+0000", 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059680519999999997 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244848+0000", 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2145999999999996e-05 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245196+0000", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034871799999999998 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246712+0000", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015162190000000001 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250197+0000", 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034843389999999999 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250250+0000", 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3035999999999998e-05 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250292+0000", 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2120999999999999e-05 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10831 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8922071460000001, 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012631310999999999, 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10831 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238822+0000", 2024-04-23T21:21:37.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7800000000000002e-07 2024-04-23T21:21:37.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238827+0000", 2024-04-23T21:21:37.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4310000000000002e-06 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238837+0000", 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4010000000000003e-06 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238843+0000", 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2079999999999997e-06 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245485+0000", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0066425499999999997 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245536+0000", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0149999999999999e-05 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245733+0000", 2024-04-23T21:21:37.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000197231 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246718+0000", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00098556699999999991 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251355+0000", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046364290000000001 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251406+0000", 2024-04-23T21:21:37.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1038999999999997e-05 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251453+0000", 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6826999999999997e-05 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10832 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8921581170000001, 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012899838, 2024-04-23T21:21:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10832 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238871+0000", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3200000000000005e-07 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238874+0000", 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4429999999999999e-06 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238883+0000", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6859999999999995e-06 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238891+0000", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9889999999999992e-06 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246016+0000", 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071248190000000001 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246065+0000", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8872999999999998e-05 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246281+0000", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000216393 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249969+0000", 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036876690000000002 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251663+0000", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016940519999999999 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251725+0000", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1818999999999999e-05 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251770+0000", 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5562999999999997e-05 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.213 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10833 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8914730990000002, 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012352699999999999, 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.214 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10833 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239556+0000", 2024-04-23T21:21:37.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239562+0000", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1610000000000002e-06 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239572+0000", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0738e-05 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239579+0000", 2024-04-23T21:21:37.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1099999999999997e-06 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246746+0000", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071668790000000001 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246798+0000", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1748e-05 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246995+0000", 2024-04-23T21:21:37.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019731800000000001 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249976+0000", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002981058 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251830+0000", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00185395 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251884+0000", 2024-04-23T21:21:37.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3171999999999998e-05 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251908+0000", 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4652999999999999e-05 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10834 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8910090349999997, 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012040969, 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.219 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10834 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8599999999999997e-07 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240034+0000", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3552e-05 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240046+0000", 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1924e-05 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240052+0000", 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-06 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247246+0000", 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071939179999999997 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247321+0000", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5110000000000004e-05 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247520+0000", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000198743 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250307+0000", 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002786462 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251964+0000", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001657309 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252019+0000", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4728000000000002e-05 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252060+0000", 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1847000000000001e-05 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.224 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10835 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8907672130000002, 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011966503999999999, 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10835 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.225 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240262+0000", 2024-04-23T21:21:37.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240268+0000", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7370000000000003e-06 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240279+0000", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1569000000000001e-05 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240287+0000", 2024-04-23T21:21:37.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.926e-06 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247891+0000", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0076038410000000001 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247988+0000", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6704000000000001e-05 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248469+0000", 2024-04-23T21:21:37.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00048112900000000001 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250313+0000", 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018440119999999999 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252122+0000", 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018091990000000001 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252178+0000", 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5603000000000003e-05 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252228+0000", 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9985000000000003e-05 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10836 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8902392240000001, 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011591181000000001, 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10836 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240790+0000", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7400000000000002e-07 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240795+0000", 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4689999999999999e-06 2024-04-23T21:21:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240803+0000", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4940000000000003e-06 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240808+0000", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9729999999999997e-06 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248916+0000", 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081076729999999993 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249008+0000", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2650000000000002e-05 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249420+0000", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041150699999999998 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250324+0000", 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00090375899999999996 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252289+0000", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019649149999999998 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252343+0000", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3977999999999997e-05 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252381+0000", 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10839 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.8892419650000001, 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010757071999999999, 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10839 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.236 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9999999999999997e-07 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241792+0000", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4460000000000003e-06 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241804+0000", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.257e-05 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241812+0000", 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3089999999999998e-06 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250357+0000", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0085456630000000002 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250606+0000", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024869900000000001 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250982+0000", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037560399999999998 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:37.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252243+0000", 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001261306 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252436+0000", 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019354800000000001 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252502+0000", 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5513999999999996e-05 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252544+0000", 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1712999999999999e-05 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.241 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:37.243 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:37.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:37.244 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:37.244+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:37.244 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-04-23T21:21:37.244 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-04-23T21:21:37.249 INFO:teuthology.orchestra.run.smithi111.stdout:1411 trunc from 0x28f3873 to 0xb5e46b3 2024-04-23T21:21:37.258 INFO:teuthology.orchestra.run.smithi111.stdout:1412 read 0x2f8b0a4 thru 0x2f973c5 (0xc322 bytes) 2024-04-23T21:21:37.258 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:37.260 INFO:teuthology.orchestra.run.smithi111.stdout:1414 trunc from 0xb5e46b3 to 0xdbe6f48 2024-04-23T21:21:37.266 INFO:teuthology.orchestra.run.smithi111.stdout:1415 read 0x5335cde thru 0x534384f (0xdb72 bytes) 2024-04-23T21:21:37.266 INFO:teuthology.orchestra.run.smithi111.stdout:1416 read 0x1384d2 thru 0x14017f (0x7cae bytes) 2024-04-23T21:21:37.266 INFO:teuthology.orchestra.run.smithi111.stdout:1417 writesame 0x1c77e23 thru 0x1c7cab6 data_size 0x4(0x4c94 bytes) 2024-04-23T21:21:37.271 INFO:teuthology.orchestra.run.smithi111.stdout:1418 write 0x444766d thru 0x44536cd (0xc061 bytes) 2024-04-23T21:21:37.275 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:37.275 INFO:teuthology.orchestra.run.smithi111.stdout:1421 read 0x6f616f2 thru 0x6f659b6 (0x42c5 bytes) 2024-04-23T21:21:37.286 INFO:teuthology.orchestra.run.smithi111.stdout:1422 trunc from 0xdbe6f48 to 0x45510c2 2024-04-23T21:21:37.308 INFO:teuthology.orchestra.run.smithi111.stdout:1423 read 0x1befb5f thru 0x1bf8a26 (0x8ec8 bytes) 2024-04-23T21:21:37.309 INFO:teuthology.orchestra.run.smithi111.stdout:1424 writesame 0x26141b5 thru 0x2618363 data_size 0x13(0x41af bytes) 2024-04-23T21:21:37.310 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:37.321 INFO:teuthology.orchestra.run.smithi111.stdout:1427 write 0x85de4a4 thru 0x85e9ec9 (0xba26 bytes) 2024-04-23T21:21:37.336 INFO:teuthology.orchestra.run.smithi111.stdout:1431 writesame 0xd8b0abc thru 0xd8bcf6c data_size 0x1(0xc4b1 bytes) 2024-04-23T21:21:37.336 INFO:teuthology.orchestra.run.smithi111.stdout:1432 write 0xb64408c thru 0xb6492a6 (0x521b bytes) 2024-04-23T21:21:37.340 INFO:teuthology.orchestra.run.smithi111.stdout:1433 trunc from 0xd8bcf6d to 0xce3324e 2024-04-23T21:21:37.343 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:37.344+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:37.346 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:37.346 INFO:teuthology.orchestra.run.smithi111.stdout:1436 punch from 0x1f93cf5 to 0x1f9a559, (0x6864 bytes) 2024-04-23T21:21:37.346 INFO:teuthology.orchestra.run.smithi111.stdout:1439 write 0x16bf4db thru 0x16c7b31 (0x8657 bytes) 2024-04-23T21:21:37.347 INFO:teuthology.orchestra.run.smithi111.stdout:1440 read 0xaed3b9d thru 0xaedf603 (0xba67 bytes) 2024-04-23T21:21:37.348 INFO:teuthology.orchestra.run.smithi111.stdout:1442 punch from 0xc24fec1 to 0xc25b5ff, (0xb73e bytes) 2024-04-23T21:21:37.348 INFO:teuthology.orchestra.run.smithi111.stdout:1444 writesame 0x580de7d thru 0x5812beb data_size 0x1(0x4d6f bytes) 2024-04-23T21:21:37.354 INFO:teuthology.orchestra.run.smithi111.stdout:1445 write 0x4a3a2b9 thru 0x4a3cbe2 (0x292a bytes) 2024-04-23T21:21:37.355 INFO:teuthology.orchestra.run.smithi111.stdout:1448 read 0x42744d thru 0x42e97e (0x7532 bytes) 2024-04-23T21:21:37.356 INFO:teuthology.orchestra.run.smithi111.stdout:1449 write 0x669e11f thru 0x66a66ff (0x85e1 bytes) 2024-04-23T21:21:37.357 INFO:teuthology.orchestra.run.smithi111.stdout:1451 write 0x4c1b68f thru 0x4c25d86 (0xa6f8 bytes) 2024-04-23T21:21:37.359 INFO:teuthology.orchestra.run.smithi111.stdout:1452 writesame 0xbca6e32 thru 0xbcb5760 data_size 0x1(0xe92f bytes) 2024-04-23T21:21:37.390 INFO:teuthology.orchestra.run.smithi111.stdout:1453 write 0x40ee3c7 thru 0x40f7a90 (0x96ca bytes) 2024-04-23T21:21:37.392 INFO:teuthology.orchestra.run.smithi111.stdout:1454 punch from 0x541a00 to 0x546782, (0x4d82 bytes) 2024-04-23T21:21:37.392 INFO:teuthology.orchestra.run.smithi111.stdout:1455 write 0xa921fb3 thru 0xa931100 (0xf14e bytes) 2024-04-23T21:21:37.393 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-04-23T21:21:37.400 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:37.400 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:37.400 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:37.400 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:37.406 INFO:teuthology.orchestra.run.smithi111.stdout:1456 trunc from 0xce3324e to 0x33aad61 2024-04-23T21:21:37.409 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:37.424 INFO:teuthology.orchestra.run.smithi111.stdout:1457 write 0x347765c thru 0x348512c (0xdad1 bytes) 2024-04-23T21:21:37.430 INFO:teuthology.orchestra.run.smithi111.stdout:1458 trunc from 0x348512d to 0x7d54eef 2024-04-23T21:21:37.436 INFO:teuthology.orchestra.run.smithi111.stdout:1459 write 0x37f9b03 thru 0x3809826 (0xfd24 bytes) 2024-04-23T21:21:37.438 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:37.438 INFO:teuthology.orchestra.run.smithi111.stdout:1461 read 0x29ae929 thru 0x29b9192 (0xa86a bytes) 2024-04-23T21:21:37.450 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:37.451+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:1462 writesame 0xe563c66 thru 0xe56c18b data_size 0x27(0x8526 bytes) 2024-04-23T21:21:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:1463 read 0xe480019 thru 0xe48c0af (0xc097 bytes) 2024-04-23T21:21:37.452 INFO:teuthology.orchestra.run.smithi111.stdout:1465 write 0x230bb41 thru 0x2312e53 (0x7313 bytes) 2024-04-23T21:21:37.453 INFO:teuthology.orchestra.run.smithi111.stdout:1466 write 0xbbe0bca thru 0xbbef680 (0xeab7 bytes) 2024-04-23T21:21:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:1467 read 0x6ac7636 thru 0x6acef2d (0x78f8 bytes) 2024-04-23T21:21:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:1468 punch from 0x69b1e21 to 0x69b918c, (0x736b bytes) 2024-04-23T21:21:37.456 INFO:teuthology.orchestra.run.smithi111.stdout:1469 read 0xa542ed9 thru 0xa54e8b5 (0xb9dd bytes) 2024-04-23T21:21:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:1470 read 0x1a52bd thru 0x1af8a7 (0xa5eb bytes) 2024-04-23T21:21:37.457 INFO:teuthology.orchestra.run.smithi111.stdout:1472 write 0xa024bc5 thru 0xa0324e6 (0xd922 bytes) 2024-04-23T21:21:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:1473 punch from 0x1202b9c to 0x120b926, (0x8d8a bytes) 2024-04-23T21:21:37.461 INFO:teuthology.orchestra.run.smithi111.stdout:1478 clone 12 order 22 su 65536 sc 8 2024-04-23T21:21:37.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-04-23T21:21:37.546 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:37.546+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:37.553 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:37.553 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:37.553 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:37.553 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:37.553 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:37.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:37.646 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:37.647+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:37.707 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3197 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2916352~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286899741999999, 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024247707, 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3197 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420534+0000", 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.074e-06 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420544+0000", 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3370000000000006e-06 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420549+0000", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9030000000000003e-06 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444634+0000", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024085517000000001 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444713+0000", 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444780+0000", 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6797000000000004e-05 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3198 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2981888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286876134, 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024356404000000002, 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3198 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420556+0000", 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420558+0000", 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420564+0000", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7089999999999999e-06 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420568+0000", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6280000000000004e-06 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444802+0000", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024234169999999999 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444844+0000", 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444912+0000", 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8367999999999993e-05 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286857181, 2024-04-23T21:21:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286829817999999, 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:21:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286815569, 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:21:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286793326, 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:21:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:21:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286727839999999, 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:21:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286702097999999, 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286678089, 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286655530000001, 2024-04-23T21:21:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:21:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:21:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286635727, 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:21:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286616241000001, 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286596442, 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:21:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286576739999999, 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286520614000001, 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286483392999999, 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:21:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:21:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.28643308, 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:21:37.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286400185, 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:21:37.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286377881, 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:21:37.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:21:37.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.286357465, 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:37.788 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:37.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:37.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:37.792 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:37.794 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-04-23T21:21:37.795 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:37.795+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:37.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:37.873 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:21:37.894 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:37.895+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:37.948 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone11 intact 2024-04-23T21:21:37.950 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:37.950 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:37.950 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:37.950 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:37.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:38.005 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:38.005+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:38.009 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-04-23T21:21:38.071 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.071 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:38.071 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:38.071 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:38.071 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.082 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:38.095 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:38.096+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:38.132 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-04-23T21:21:38.132 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.132 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.132 INFO:teuthology.orchestra.run.smithi111.stdout:1482 punch from 0x4fed3ec to 0x4ffa63c, (0xd250 bytes) 2024-04-23T21:21:38.135 INFO:teuthology.orchestra.run.smithi111.stdout:1483 write 0xc544b8d thru 0xc551f48 (0xd3bc bytes) 2024-04-23T21:21:38.141 INFO:teuthology.orchestra.run.smithi111.stdout:1485 write 0x37c0cdf thru 0x37cde72 (0xd194 bytes) 2024-04-23T21:21:38.152 INFO:teuthology.orchestra.run.smithi111.stdout:1486 trunc from 0xe56c18c to 0xa16f344 2024-04-23T21:21:38.183 INFO:teuthology.orchestra.run.smithi111.stdout:1487 punch from 0x40ffb62 to 0x410c675, (0xcb13 bytes) 2024-04-23T21:21:38.191 INFO:teuthology.orchestra.run.smithi111.stdout:1489 write 0xc0f67de thru 0xc1067d0 (0xfff3 bytes) 2024-04-23T21:21:38.194 INFO:teuthology.orchestra.run.smithi111.stdout:1490 read 0x4a5044d thru 0x4a5ad7b (0xa92f bytes) 2024-04-23T21:21:38.194 INFO:teuthology.orchestra.run.smithi111.stdout:1491 writesame 0xb77ce68 thru 0xb77faaf data_size 0x34(0x2c48 bytes) 2024-04-23T21:21:38.196 INFO:teuthology.orchestra.run.smithi111.stdout:1492 punch from 0x53ddfa2 to 0x53ec0d7, (0xe135 bytes) 2024-04-23T21:21:38.197 INFO:teuthology.orchestra.run.smithi111.stdout:1494 read 0xb5699d4 thru 0xb5740fd (0xa72a bytes) 2024-04-23T21:21:38.197 INFO:teuthology.orchestra.run.smithi111.stdout:1495 writesame 0x751cfca thru 0x75292a7 data_size 0x1(0xc2de bytes) 2024-04-23T21:21:38.197 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:38.197+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:38.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-04-23T21:21:38.229 INFO:teuthology.orchestra.run.smithi111.stdout:1496 write 0xecb0eec thru 0xecbf3a2 (0xe4b7 bytes) 2024-04-23T21:21:38.232 INFO:teuthology.orchestra.run.smithi111.stdout:1497 read 0x45f50fd thru 0x46021ba (0xd0be bytes) 2024-04-23T21:21:38.233 INFO:teuthology.orchestra.run.smithi111.stdout:1498 write 0x1e128e8 thru 0x1e20829 (0xdf42 bytes) 2024-04-23T21:21:38.245 INFO:teuthology.orchestra.run.smithi111.stdout:1500 write 0x38e031d thru 0x38eb72f (0xb413 bytes) 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8159448880000006, 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.247 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8159220769999997, 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.252 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:38.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:21:38.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8159024670000008, 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.256 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8158836039999997, 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.260 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8158643090000002, 2024-04-23T21:21:38.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.268 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8158377479999999, 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:21:38.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:21:38.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8157965179999991, 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.273 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:21:38.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:21:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8157627910000009, 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8157393820000003, 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8157061320000007, 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8156827389999997, 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:21:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:21:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8156621899999994, 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:21:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8156364299999996, 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:21:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:21:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8147984140000002, 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:21:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:21:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:21:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8147750830000007, 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:21:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:21:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:21:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8138877900000008, 2024-04-23T21:21:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:21:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8136542099999993, 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:21:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:21:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8135375119999999, 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:21:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8131630980000004, 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:21:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.8118062130000006, 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:21:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:21:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.334 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.335 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.335 INFO:teuthology.orchestra.run.smithi111.stdout:1502 punch from 0xe451b49 to 0xe452a34, (0xeeb bytes) 2024-04-23T21:21:38.335 INFO:teuthology.orchestra.run.smithi111.stdout:1503 write 0x2dd7dca thru 0x2ddaa70 (0x2ca7 bytes) 2024-04-23T21:21:38.335 INFO:teuthology.orchestra.run.smithi111.stdout:1504 write 0x75d0f3e thru 0x75dad27 (0x9dea bytes) 2024-04-23T21:21:38.336 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:38.337 INFO:teuthology.orchestra.run.smithi111.stdout:1506 read 0x66b8916 thru 0x66ba2b1 (0x199c bytes) 2024-04-23T21:21:38.337 INFO:teuthology.orchestra.run.smithi111.stdout:1507 writesame 0x7d6e639 thru 0x7d767b3 data_size 0x57(0x817b bytes) 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1508 trunc from 0xecbf3a3 to 0xe543124 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1509 trunc from 0xe543124 to 0x30269bf 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1512 read 0x2f14b50 thru 0x2f207a7 (0xbc58 bytes) 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1513 write 0x6ae47db thru 0x6aeb399 (0x6bbf bytes) 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1516 read 0x2a54006 thru 0x2a54c1e (0xc19 bytes) 2024-04-23T21:21:38.338 INFO:teuthology.orchestra.run.smithi111.stdout:1518 trunc from 0x6aeb39a to 0x53446d6 2024-04-23T21:21:38.338 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:38.336+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:38.347 INFO:teuthology.orchestra.run.smithi111.stdout:1519 read 0x3416d07 thru 0x341789a (0xb94 bytes) 2024-04-23T21:21:38.347 INFO:teuthology.orchestra.run.smithi111.stdout:1520 read 0x2b5d6f2 thru 0x2b621c8 (0x4ad7 bytes) 2024-04-23T21:21:38.349 INFO:teuthology.orchestra.run.smithi111.stdout:1521 read 0x16b093c thru 0x16be7ba (0xde7f bytes) 2024-04-23T21:21:38.349 INFO:teuthology.orchestra.run.smithi111.stdout:1522 write 0x3c665c8 thru 0x3c70e69 (0xa8a2 bytes) 2024-04-23T21:21:38.353 INFO:teuthology.orchestra.run.smithi111.stdout:1523 writesame 0xa06972 thru 0xa12285 data_size 0xa(0xb914 bytes) 2024-04-23T21:21:38.378 INFO:teuthology.orchestra.run.smithi111.stdout:1524 writesame 0xda07e9d thru 0xda16ebe data_size 0x7(0xf022 bytes) 2024-04-23T21:21:38.378 INFO:teuthology.orchestra.run.smithi111.stdout:1526 read 0xb7fc08e thru 0xb7fd613 (0x1586 bytes) 2024-04-23T21:21:38.378 INFO:teuthology.orchestra.run.smithi111.stdout:1527 write 0xa7a5ce3 thru 0xa7a60ef (0x40d bytes) 2024-04-23T21:21:38.380 INFO:teuthology.orchestra.run.smithi111.stdout:1528 read 0xb9a57d2 thru 0xb9b3102 (0xd931 bytes) 2024-04-23T21:21:38.394 INFO:teuthology.orchestra.run.smithi111.stdout:1529 trunc from 0xda16ebf to 0x2623aa0 2024-04-23T21:21:38.407 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.411 INFO:teuthology.orchestra.run.smithi111.stdout:1531 trunc from 0x2623aa0 to 0x572827d 2024-04-23T21:21:38.429 INFO:teuthology.orchestra.run.smithi111.stdout:1532 writesame 0xb8c542e thru 0xb8d4f48 data_size 0x1(0xfb1b bytes) 2024-04-23T21:21:38.434 INFO:teuthology.orchestra.run.smithi111.stdout:1534 trunc from 0xb8d4f49 to 0x7a2586d 2024-04-23T21:21:38.436 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:38.436+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:38.439 INFO:teuthology.orchestra.run.smithi111.stdout:1535 write 0x77ffbe2 thru 0x780b6f0 (0xbb0f bytes) 2024-04-23T21:21:38.444 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.454 INFO:teuthology.orchestra.run.smithi111.stdout:1537 write 0xb4cc7b5 thru 0xb4da383 (0xdbcf bytes) 2024-04-23T21:21:38.458 INFO:teuthology.orchestra.run.smithi111.stdout:1538 writesame 0x221c37d thru 0x22252bb data_size 0x1(0x8f3f bytes) 2024-04-23T21:21:38.465 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.465 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:38.465 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:38.465 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:38.481 INFO:teuthology.orchestra.run.smithi111.stdout:1539 trunc from 0xb4da384 to 0x10f3093 2024-04-23T21:21:38.516 INFO:teuthology.orchestra.run.smithi111.stdout:1540 read 0xbc512 thru 0xbd5c6 (0x10b5 bytes) 2024-04-23T21:21:38.517 INFO:teuthology.orchestra.run.smithi111.stdout:1541 punch from 0xd38094 to 0xd3dc1c, (0x5b88 bytes) 2024-04-23T21:21:38.517 INFO:teuthology.orchestra.run.smithi111.stdout:1542 read 0x97765e thru 0x97bf7d (0x4920 bytes) 2024-04-23T21:21:38.517 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.529 INFO:teuthology.orchestra.run.smithi111.stdout:1544 write 0x6a1dbf3 thru 0x6a26944 (0x8d52 bytes) 2024-04-23T21:21:38.537 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:38.537+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:38.542 INFO:teuthology.orchestra.run.smithi111.stdout:1545 write 0x9a7eecc thru 0x9a871df (0x8314 bytes) 2024-04-23T21:21:38.545 INFO:teuthology.orchestra.run.smithi111.stdout:1546 write 0x238a08e thru 0x238ce13 (0x2d86 bytes) 2024-04-23T21:21:38.548 INFO:teuthology.orchestra.run.smithi111.stdout:1547 read 0x52be84a thru 0x52c79ba (0x9171 bytes) 2024-04-23T21:21:38.548 INFO:teuthology.orchestra.run.smithi111.stdout:1548 punch from 0x16ac150 to 0x16b91d7, (0xd087 bytes) 2024-04-23T21:21:38.548 INFO:teuthology.orchestra.run.smithi111.stdout:1549 writesame 0x906f028 thru 0x906f5e0 data_size 0x1(0x5b9 bytes) 2024-04-23T21:21:38.551 INFO:teuthology.orchestra.run.smithi111.stdout:1550 read 0x87d288e thru 0x87d4dd9 (0x254c bytes) 2024-04-23T21:21:38.551 INFO:teuthology.orchestra.run.smithi111.stdout:1551 punch from 0x1337c2f to 0x1343ce7, (0xc0b8 bytes) 2024-04-23T21:21:38.560 INFO:teuthology.orchestra.run.smithi111.stdout:1552 writesame 0xc3a2f66 thru 0xc3aaa70 data_size 0x1(0x7b0b bytes) 2024-04-23T21:21:38.560 INFO:teuthology.orchestra.run.smithi111.stdout:1553 write 0x2b8c73f thru 0x2b92270 (0x5b32 bytes) 2024-04-23T21:21:38.562 INFO:teuthology.orchestra.run.smithi111.stdout:1555 read 0x444ad3c thru 0x4455b5f (0xae24 bytes) 2024-04-23T21:21:38.562 INFO:teuthology.orchestra.run.smithi111.stdout:1556 write 0x6625aed thru 0x66263a0 (0x8b4 bytes) 2024-04-23T21:21:38.563 INFO:teuthology.orchestra.run.smithi111.stdout:1557 read 0x1307235 thru 0x13120f1 (0xaebd bytes) 2024-04-23T21:21:38.564 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.564 INFO:teuthology.orchestra.run.smithi111.stdout:1559 writesame 0x31887b7 thru 0x3194acb data_size 0x5d(0xc315 bytes) 2024-04-23T21:21:38.566 INFO:teuthology.orchestra.run.smithi111.stdout:1560 read 0x2299aec thru 0x22a57b7 (0xbccc bytes) 2024-04-23T21:21:38.574 INFO:teuthology.orchestra.run.smithi111.stdout:1561 write 0xe7bdfee thru 0xe7be62a (0x63d bytes) 2024-04-23T21:21:38.578 INFO:teuthology.orchestra.run.smithi111.stdout:1563 read 0x4bfe1ae thru 0x4c001c6 (0x2019 bytes) 2024-04-23T21:21:38.578 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.578 INFO:teuthology.orchestra.run.smithi111.stdout:1566 read 0x57272a thru 0x582267 (0xfb3e bytes) 2024-04-23T21:21:38.578 INFO:teuthology.orchestra.run.smithi111.stdout:1568 read 0xd76c34f thru 0xd772732 (0x63e4 bytes) 2024-04-23T21:21:38.590 INFO:teuthology.orchestra.run.smithi111.stdout:1570 trunc from 0xe7be62b to 0x4df4968 2024-04-23T21:21:38.599 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.599 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:38.599 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:38.599 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:38.599 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.610 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:38.612 INFO:teuthology.orchestra.run.smithi111.stdout:1571 write 0x79a6193 thru 0x79b58c0 (0xf72e bytes) 2024-04-23T21:21:38.620 INFO:teuthology.orchestra.run.smithi111.stdout:1574 trunc from 0x79b58c1 to 0xbc09992 2024-04-23T21:21:38.627 INFO:teuthology.orchestra.run.smithi111.stdout:1576 writesame 0x27cac43 thru 0x27d68a3 data_size 0x19(0xbc61 bytes) 2024-04-23T21:21:38.633 INFO:teuthology.orchestra.run.smithi111.stdout:1577 read 0x6bde26a thru 0x6bdecc5 (0xa5c bytes) 2024-04-23T21:21:38.633 INFO:teuthology.orchestra.run.smithi111.stdout:1578 punch from 0x27a24d7 to 0x27a3a01, (0x152a bytes) 2024-04-23T21:21:38.638 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:38.638+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:38.643 INFO:teuthology.orchestra.run.smithi111.stdout:1579 trunc from 0xbc09992 to 0x35ff37e 2024-04-23T21:21:38.656 INFO:teuthology.orchestra.run.smithi111.stdout:1580 read 0x1e86e11 thru 0x1e9087f (0x9a6f bytes) 2024-04-23T21:21:38.656 INFO:teuthology.orchestra.run.smithi111.stdout:1582 read 0x2eb8e77 thru 0x2ebb68e (0x2818 bytes) 2024-04-23T21:21:38.666 INFO:teuthology.orchestra.run.smithi111.stdout:1584 write 0x8071cce thru 0x807fcfe (0xe031 bytes) 2024-04-23T21:21:38.669 INFO:teuthology.orchestra.run.smithi111.stdout:1585 write 0x61fecae thru 0x6200495 (0x17e8 bytes) 2024-04-23T21:21:38.673 INFO:teuthology.orchestra.run.smithi111.stdout:1589 read 0x5fc491a thru 0x5fc767c (0x2d63 bytes) 2024-04-23T21:21:38.673 INFO:teuthology.orchestra.run.smithi111.stdout:1590 read 0x7a8a280 thru 0x7a99e85 (0xfc06 bytes) 2024-04-23T21:21:38.673 INFO:teuthology.orchestra.run.smithi111.stdout:1592 punch from 0x4376589 to 0x4384a79, (0xe4f0 bytes) 2024-04-23T21:21:38.673 INFO:teuthology.orchestra.run.smithi111.stdout:1594 punch from 0x2414142 to 0x24174e2, (0x33a0 bytes) 2024-04-23T21:21:38.674 INFO:teuthology.orchestra.run.smithi111.stdout:1596 read 0x4f1694 thru 0x4f3baa (0x2517 bytes) 2024-04-23T21:21:38.674 INFO:teuthology.orchestra.run.smithi111.stdout:1598 write 0x1f89b1c thru 0x1f90a56 (0x6f3b bytes) 2024-04-23T21:21:38.675 INFO:teuthology.orchestra.run.smithi111.stdout:1599 writesame 0x47660f8 thru 0x4772a61 data_size 0x1d(0xc96a bytes) 2024-04-23T21:21:38.680 INFO:teuthology.orchestra.run.smithi111.stdout:1600 punch from 0x554656a to 0x554d957, (0x73ed bytes) 2024-04-23T21:21:38.680 INFO:teuthology.orchestra.run.smithi111.stdout:1601 read 0x7ea78dd thru 0x7eacf39 (0x565d bytes) 2024-04-23T21:21:38.689 INFO:teuthology.orchestra.run.smithi111.stdout:1602 write 0xae8584e thru 0xae87fd6 (0x2789 bytes) 2024-04-23T21:21:38.691 INFO:teuthology.orchestra.run.smithi111.stdout:1603 read 0x25781a0 thru 0x2583106 (0xaf67 bytes) 2024-04-23T21:21:38.692 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.694 INFO:teuthology.orchestra.run.smithi111.stdout:1605 trunc from 0xae87fd7 to 0x8e1165b 2024-04-23T21:21:38.707 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.707 INFO:teuthology.orchestra.run.smithi111.stdout:1613 read 0x8903108 thru 0x890f7d0 (0xc6c9 bytes) 2024-04-23T21:21:38.708 INFO:teuthology.orchestra.run.smithi111.stdout:1614 read 0x2c9f3e3 thru 0x2ca512f (0x5d4d bytes) 2024-04-23T21:21:38.708 INFO:teuthology.orchestra.run.smithi111.stdout:1615 punch from 0x312cdb8 to 0x3138421, (0xb669 bytes) 2024-04-23T21:21:38.708 INFO:teuthology.orchestra.run.smithi111.stdout:1617 read 0x54145fe thru 0x541eb35 (0xa538 bytes) 2024-04-23T21:21:38.708 INFO:teuthology.orchestra.run.smithi111.stdout:1619 punch from 0x9161fb to 0x91bb39, (0x593e bytes) 2024-04-23T21:21:38.708 INFO:teuthology.orchestra.run.smithi111.stdout:1620 trunc from 0x8e1165b to 0x897056c 2024-04-23T21:21:38.714 INFO:teuthology.orchestra.run.smithi111.stdout:1621 trunc from 0x897056c to 0x84b9f66 2024-04-23T21:21:38.719 INFO:teuthology.orchestra.run.smithi111.stdout:1622 punch from 0x1361f01 to 0x1362969, (0xa68 bytes) 2024-04-23T21:21:38.719 INFO:teuthology.orchestra.run.smithi111.stdout:1624 write 0x78e21a1 thru 0x78f1a2c (0xf88c bytes) 2024-04-23T21:21:38.722 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:38.722 INFO:teuthology.orchestra.run.smithi111.stdout:1626 punch from 0x839b912 to 0x83aa160, (0xe84e bytes) 2024-04-23T21:21:38.722 INFO:teuthology.orchestra.run.smithi111.stdout:1627 write 0x5f95f29 thru 0x5f9a0f4 (0x41cc bytes) 2024-04-23T21:21:38.726 INFO:teuthology.orchestra.run.smithi111.stdout:1628 read 0x77d53da thru 0x77e08a5 (0xb4cc bytes) 2024-04-23T21:21:38.726 INFO:teuthology.orchestra.run.smithi111.stdout:1629 punch from 0x5d6e364 to 0x5d7d77f, (0xf41b bytes) 2024-04-23T21:21:38.727 INFO:teuthology.orchestra.run.smithi111.stdout:1630 clone 13 order 25 su 4194304 sc 9 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:7796 3.9 e27/18 3:9154b378:::rbd_data.111830ac55a1.0000000000000028:head v 27'28, mlcod=27'26)", 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:38.728 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.7351263499999998, 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046354710000000004, 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:38.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.68e-07 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993123+0000", 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993133+0000", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0240000000000003e-06 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993137+0000", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993191+0000", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3300000000000001e-05 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993213+0000", 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.23e-05 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.997710+0000", 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0044970469999999997 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.997753+0000", 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3373e-05 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:10838 3.9 e31/18 3:9387dc3a:::rbd_data.111823663ad5.0000000000000001:head v 31'44, mlcod=30'42)", 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.4847725699999996, 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049523199999999996, 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243473+0000", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3459999999999999e-06 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244449+0000", 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00097653599999999996 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244463+0000", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3699e-05 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244471+0000", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9810000000000003e-06 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244528+0000", 2024-04-23T21:21:38.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7246e-05 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244551+0000", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2331999999999999e-05 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.248369+0000", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0038185010000000002 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.248424+0000", 2024-04-23T21:21:38.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4679000000000003e-05 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:13154 3.0 3:0277b7fa:::rbd_data.11188429226.000000000000000d:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3786381940000001, 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0062056239999999999, 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 13154 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349608+0000", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3539999999999998e-06 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349617+0000", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6959999999999994e-06 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349636+0000", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9392e-05 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349647+0000", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0199e-05 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349701+0000", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4910000000000001e-05 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349757+0000", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5605e-05 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:38.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355661+0000", 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005903686 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355723+0000", 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2292000000000006e-05 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355812+0000", 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8490000000000001e-05 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14425 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 3342336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42387373900000003, 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004649355, 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14425 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.742 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304371+0000", 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6100000000000001e-07 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304373+0000", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3120000000000001e-06 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304380+0000", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.917e-06 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304385+0000", 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1680000000000003e-06 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.308891+0000", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0045060159999999998 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.308941+0000", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9360000000000002e-05 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309020+0000", 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9021000000000001e-05 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14426 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 3866624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42385213999999999, 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004767777, 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14426 2024-04-23T21:21:38.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3799999999999998e-07 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304394+0000", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0020000000000001e-06 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304402+0000", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8690000000000004e-06 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304407+0000", 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0930000000000002e-06 2024-04-23T21:21:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309045+0000", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046372829999999999 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309088+0000", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3841000000000003e-05 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309160+0000", 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1250999999999999e-05 2024-04-23T21:21:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14427 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2293760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42383020900000001, 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004892179, 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14427 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1900000000000003e-07 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304416+0000", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9819999999999998e-06 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304430+0000", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.358e-05 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304444+0000", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3599e-05 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309185+0000", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047414550000000003 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309231+0000", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5896999999999998e-05 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309306+0000", 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5147000000000004e-05 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14493 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 4128768~25853] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42246773500000001, 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047371679999999999, 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14493 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305777+0000", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305781+0000", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.856e-06 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305791+0000", 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5060000000000001e-06 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305797+0000", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5139999999999998e-06 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310152+0000", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0043545600000000004 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310194+0000", 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2179999999999999e-05 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310514+0000", 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031975299999999999 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14494 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2555904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42243746399999998, 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048527910000000004, 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14494 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3200000000000005e-07 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305809+0000", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2170000000000001e-06 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305817+0000", 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2320000000000004e-06 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305822+0000", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4e-06 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310542+0000", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047198810000000004 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310586+0000", 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4011999999999998e-05 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310659+0000", 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3417000000000005e-05 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14495 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 3080192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42241536400000002, 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049765809999999999, 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14495 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0200000000000003e-07 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305831+0000", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305839+0000", 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305844+0000", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1730000000000003e-06 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310684+0000", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048400689999999998 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310728+0000", 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4002000000000003e-05 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310805+0000", 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7349999999999996e-05 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14496 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 3604480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42239360300000001, 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050903179999999999, 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14496 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4799999999999998e-07 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305853+0000", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305859+0000", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2330000000000003e-06 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305872+0000", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2296999999999999e-05 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310828+0000", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004956645 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310869+0000", 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0229999999999999e-05 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310941+0000", 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2296e-05 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14497 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.42235139700000002, 2024-04-23T21:21:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051725909999999998, 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14497 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305896+0000", 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8590000000000001e-06 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305904+0000", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5340000000000002e-06 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305915+0000", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.113e-05 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310955+0000", 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005040615 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310992+0000", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6785000000000001e-05 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.311065+0000", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3057000000000002e-05 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14965 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.23396978800000001, 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046185159999999996, 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14965 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494275+0000", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8700000000000001e-07 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494277+0000", 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0490000000000002e-06 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494283+0000", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494287+0000", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8099999999999997e-06 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498665+0000", 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0043777579999999998 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498747+0000", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1791000000000003e-05 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498893+0000", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000145878 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14966 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.233950884, 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047961180000000003, 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14966 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494294+0000", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494295+0000", 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8029999999999999e-06 2024-04-23T21:21:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494301+0000", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6869999999999999e-06 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494306+0000", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5639999999999998e-06 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498930+0000", 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046239940000000002 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498978+0000", 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8266999999999997e-05 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499089+0000", 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011142299999999999 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15024 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.23350986000000001, 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048110970000000003, 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15024 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494735+0000", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4400000000000001e-07 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494737+0000", 2024-04-23T21:21:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494744+0000", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0890000000000002e-06 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494749+0000", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2560000000000002e-06 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499335+0000", 2024-04-23T21:21:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0045854670000000002 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499415+0000", 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0197000000000005e-05 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499545+0000", 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013021300000000001 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15025 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 1507328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.23348719500000001, 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050516770000000001, 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15025 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.65e-07 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494759+0000", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9850000000000001e-06 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494767+0000", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2179999999999999e-06 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494773+0000", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7849999999999997e-06 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499588+0000", 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048144540000000001 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499677+0000", 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9622000000000004e-05 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499809+0000", 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013124799999999999 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15026 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 2031616~52822] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.233464017, 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005573031, 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15026 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494781+0000", 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494795+0000", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4107999999999999e-05 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494802+0000", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6229999999999997e-06 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494816+0000", 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3553999999999999e-05 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499863+0000", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050470280000000003 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499961+0000", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8034000000000004e-05 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500353+0000", 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000392273 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15027 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.233420667, 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0058122069999999998, 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15027 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494824+0000", 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5600000000000001e-07 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494826+0000", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2010000000000002e-06 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494834+0000", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2199999999999992e-06 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494839+0000", 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.011e-06 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500400+0000", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0055606379999999997 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500495+0000", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4664e-05 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500636+0000", 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014101700000000001 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15028 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.233398512, 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0059530670000000003, 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15028 2024-04-23T21:21:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494848+0000", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9010000000000001e-06 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494855+0000", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6030000000000001e-06 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494859+0000", 2024-04-23T21:21:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.831e-06 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500667+0000", 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0058079400000000002 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500710+0000", 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2803000000000003e-05 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500799+0000", 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8438999999999998e-05 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15029 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1507328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.23337834199999999, 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.006189295, 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15029 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494870+0000", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5319999999999998e-06 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494877+0000", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7479999999999993e-06 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494883+0000", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3029999999999999e-06 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500836+0000", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0059528910000000001 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500917+0000", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.14e-05 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501055+0000", 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013802199999999999 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15030 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2031616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.233354063, 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0063112130000000004, 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15030 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494891+0000", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1600000000000001e-07 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494893+0000", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0880000000000002e-06 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494905+0000", 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1875999999999999e-05 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494917+0000", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2024999999999999e-05 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501074+0000", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0061574120000000001 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501127+0000", 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2602000000000002e-05 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501201+0000", 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4593999999999998e-05 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:38.811 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.812 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:38.813+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:38.813 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:38.913 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:38.914+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:38.954 INFO:teuthology.orchestra.run.smithi111.stdout:flattening image image_client.0-clone12 2024-04-23T21:21:38.959 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:38.960 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:38.960 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:38.960 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:38.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:39.015 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:39.016+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:39.069 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-04-23T21:21:39.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:39.078 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:39.079 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:39.079 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:39.079 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:39.079 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:39.091 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:39.116 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:39.117+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:39.210 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:39.211 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:39.211 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:39.217 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:39.218+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:39.230 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:39.230 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:39.263 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:39.263 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7260 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6684672~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8115787680000004, 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026061528, 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7260 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451672+0000", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3179999999999999e-06 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451676+0000", 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4199999999999999e-06 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451680+0000", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6279999999999998e-06 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477564+0000", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025884564999999998 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477641+0000", 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7300999999999997e-05 2024-04-23T21:21:39.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477732+0000", 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0884999999999995e-05 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7261 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6881280~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8115645980000004, 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026239373, 2024-04-23T21:21:39.268 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7261 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5100000000000001e-07 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451688+0000", 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.255e-06 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451693+0000", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8540000000000002e-06 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451698+0000", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4610000000000001e-06 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477761+0000", 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026062980999999999 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477836+0000", 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5496999999999999e-05 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477924+0000", 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7973999999999995e-05 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.272 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7262 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7077888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8115466930000004, 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026422265, 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7262 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9200000000000002e-07 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451705+0000", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8250000000000001e-06 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451712+0000", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.759e-06 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451716+0000", 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2889999999999999e-06 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477952+0000", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026235967999999998 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478032+0000", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0296999999999994e-05 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478125+0000", 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2734999999999999e-05 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7263 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7798784~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8115259609999992, 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026605936, 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7263 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452459+0000", 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00073525200000000004 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452464+0000", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8940000000000001e-06 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452474+0000", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0454e-05 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452481+0000", 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7469999999999999e-06 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478153+0000", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025672186 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478236+0000", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3061999999999999e-05 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478329+0000", 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3341e-05 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.280 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7265 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8192000~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8107360440000004, 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026201490000000001, 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.281 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7265 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452514+0000", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7899999999999999e-07 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452516+0000", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452522+0000", 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.088e-06 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452527+0000", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9420000000000003e-06 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478555+0000", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026027557 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478630+0000", 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4892000000000001e-05 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478715+0000", 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5436e-05 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7266 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8388608~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8107166229999994, 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026420895999999999, 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.285 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7266 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452535+0000", 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7990000000000001e-06 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452541+0000", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4670000000000003e-06 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452545+0000", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2089999999999999e-06 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478797+0000", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026252622 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478883+0000", 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5390999999999996e-05 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478954+0000", 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1044000000000006e-05 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7267 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9109504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8106979219999992, 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026543255000000002, 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.289 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7267 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.41e-07 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452554+0000", 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.041e-06 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452560+0000", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1179999999999997e-06 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452565+0000", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5020000000000003e-06 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478989+0000", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026423965000000001 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479040+0000", 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1697000000000003e-05 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479095+0000", 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4591e-05 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7268 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9306112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8106795059999996, 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026804591999999999, 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.293 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7268 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452572+0000", 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8419999999999999e-06 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452578+0000", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3550000000000004e-06 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452582+0000", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2780000000000004e-06 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479186+0000", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026603907 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479275+0000", 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8918999999999993e-05 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479375+0000", 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9942e-05 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7269 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9502720~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8106616350000007, 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026998115, 2024-04-23T21:21:39.297 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7269 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452611+0000", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2425e-05 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452617+0000", 2024-04-23T21:21:39.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7580000000000003e-06 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452624+0000", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7109999999999998e-06 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479405+0000", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026780531999999999 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479498+0000", 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3045000000000001e-05 2024-04-23T21:21:39.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479586+0000", 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8288999999999994e-05 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7270 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9699328~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8106177859999999, 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027156907000000001, 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.301 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7270 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452635+0000", 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.187e-06 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452640+0000", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7910000000000001e-06 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452645+0000", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8289999999999998e-06 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479616+0000", 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026971222 2024-04-23T21:21:39.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479697+0000", 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0463000000000005e-05 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479789+0000", 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1810000000000004e-05 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.305 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7271 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10420224~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.810597413, 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027566067, 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7271 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452653+0000", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452655+0000", 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0559999999999999e-06 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452660+0000", 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6289999999999998e-06 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452665+0000", 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6430000000000003e-06 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480043+0000", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027378070000000001 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480130+0000", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6879999999999995e-05 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480218+0000", 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8357999999999997e-05 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7388 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6717440~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8045918150000002, 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026122209, 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7388 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.310 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5200000000000002e-07 2024-04-23T21:21:39.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458660+0000", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5719999999999999e-06 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458665+0000", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6609999999999997e-06 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458670+0000", 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3889999999999998e-06 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484666+0000", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025996717999999999 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484724+0000", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7222999999999997e-05 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484780+0000", 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6193999999999998e-05 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.313 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7389 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6914048~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8045742830000009, 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026331738, 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.314 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7389 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458676+0000", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7500000000000001e-07 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458678+0000", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6970000000000002e-06 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458684+0000", 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5709999999999997e-06 2024-04-23T21:21:39.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458688+0000", 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6099999999999999e-06 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484821+0000", 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026132275 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484908+0000", 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7822e-05 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485007+0000", 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8487999999999996e-05 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7390 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7110656~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.318 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8045555699999998, 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026475571, 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7390 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:39.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458702+0000", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458708+0000", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0100000000000001e-06 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458712+0000", 2024-04-23T21:21:39.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485062+0000", 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026349392999999999 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485112+0000", 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0466000000000001e-05 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485170+0000", 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7160999999999997e-05 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7391 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7831552~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8045309799999991, 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026684876999999999, 2024-04-23T21:21:39.323 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7391 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7800000000000002e-07 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458722+0000", 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.587e-06 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458728+0000", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5189999999999998e-06 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458733+0000", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7140000000000001e-06 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485207+0000", 2024-04-23T21:21:39.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026474393999999998 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485300+0000", 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2644000000000003e-05 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485403+0000", 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010364100000000001 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.327 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7392 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8028160~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8045080359999996, 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026904931999999999, 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7392 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458742+0000", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5999999999999999e-07 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458744+0000", 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8509999999999999e-06 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458749+0000", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6119999999999998e-06 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458755+0000", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1440000000000002e-06 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485459+0000", 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026704828 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485548+0000", 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8985000000000003e-05 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485646+0000", 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8052000000000002e-05 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7393 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8224768~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8044892810000004, 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027031339000000001, 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7393 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:39.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460437+0000", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001677053 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460442+0000", 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:39.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460454+0000", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1704e-05 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460461+0000", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4909999999999999e-06 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485685+0000", 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025223992000000001 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485736+0000", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0633999999999997e-05 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485792+0000", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5653999999999999e-05 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7397 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9535488~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:39.335 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8026837709999999, 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026115167000000002, 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7397 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460568+0000", 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1459999999999999e-06 2024-04-23T21:21:39.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460574+0000", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6720000000000001e-06 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460579+0000", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486493+0000", 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025914311999999998 2024-04-23T21:21:39.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486582+0000", 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9132e-05 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486681+0000", 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8956000000000006e-05 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7398 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9732096~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.339 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8026648529999996, 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026253736, 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7398 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:39.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460587+0000", 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.035e-06 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460593+0000", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6670000000000001e-06 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460597+0000", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4789999999999998e-06 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486723+0000", 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026126018000000001 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486779+0000", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5445000000000002e-05 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486838+0000", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9704000000000003e-05 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7399 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10452992~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8026466019999994, 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026351192999999998, 2024-04-23T21:21:39.343 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7399 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:39.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5699999999999998e-07 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460605+0000", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.781e-06 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460611+0000", 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5509999999999999e-06 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460615+0000", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5619999999999997e-06 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486864+0000", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026248947000000002 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486911+0000", 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-05 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:39.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486954+0000", 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2664999999999999e-05 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:39.347 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:39.348 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:39.348 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:39.349+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:39.348 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:39.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:39.350 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:39.350 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:39.351 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:39.351 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:39.351 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:39.351 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:39.352 INFO:teuthology.orchestra.run.smithi111.stdout:1632 read 0xa1e0d5 thru 0xa2368f (0x55bb bytes) 2024-04-23T21:21:39.352 INFO:teuthology.orchestra.run.smithi111.stdout:1633 write 0x4526922 thru 0x4534cb8 (0xe397 bytes) 2024-04-23T21:21:39.352 INFO:teuthology.orchestra.run.smithi111.stdout:1634 write 0x170dfd7 thru 0x1713bf3 (0x5c1d bytes) 2024-04-23T21:21:39.352 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:39.354 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:39.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:39.354 INFO:teuthology.orchestra.run.smithi111.stdout:1635 write 0xe243b9f thru 0xe247ea2 (0x4304 bytes) 2024-04-23T21:21:39.354 INFO:teuthology.orchestra.run.smithi111.stdout:1636 writesame 0xcf56103 thru 0xcf63fe8 data_size 0x67(0xdee6 bytes) 2024-04-23T21:21:39.356 INFO:teuthology.orchestra.run.smithi111.stdout:1638 write 0xa00f2ba thru 0xa018ced (0x9a34 bytes) 2024-04-23T21:21:39.360 INFO:teuthology.orchestra.run.smithi111.stdout:1639 read 0xde086cc thru 0xde089e8 (0x31d bytes) 2024-04-23T21:21:39.360 INFO:teuthology.orchestra.run.smithi111.stdout:1640 punch from 0x18005d7 to 0x180e6bf, (0xe0e8 bytes) 2024-04-23T21:21:39.360 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:39.360 INFO:teuthology.orchestra.run.smithi111.stdout:1643 write 0xc409690 thru 0xc40cd7f (0x36f0 bytes) 2024-04-23T21:21:39.368 INFO:teuthology.orchestra.run.smithi111.stdout:1645 trunc from 0xe247ea3 to 0x8021dbb 2024-04-23T21:21:39.387 INFO:teuthology.orchestra.run.smithi111.stdout:1646 writesame 0x56b095c thru 0x56bb386 data_size 0xd(0xaa2b bytes) 2024-04-23T21:21:39.389 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:39.393 INFO:teuthology.orchestra.run.smithi111.stdout:1648 trunc from 0x8021dbb to 0xb66ae3c 2024-04-23T21:21:39.404 INFO:teuthology.orchestra.run.smithi111.stdout:1650 write 0xb6aab83 thru 0xb6b8c05 (0xe083 bytes) 2024-04-23T21:21:39.405 INFO:teuthology.orchestra.run.smithi111.stdout:1651 write 0x3f33d5c thru 0x3f33f6a (0x20f bytes) 2024-04-23T21:21:39.407 INFO:teuthology.orchestra.run.smithi111.stdout:1652 flatten 2024-04-23T21:21:39.412 INFO:teuthology.orchestra.run.smithi111.stdout:1653 punch from 0x4fbb1ce to 0x4fc1bbb, (0x69ed bytes) 2024-04-23T21:21:39.412 INFO:teuthology.orchestra.run.smithi111.stdout:1654 write 0x19f8fd5 thru 0x19fe45f (0x548b bytes) 2024-04-23T21:21:39.414 INFO:teuthology.orchestra.run.smithi111.stdout:1656 clone 14 order 20 su 16384 sc 5 2024-04-23T21:21:39.449 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:39.450+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:39.461 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:39.461 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:39.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:39.550 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:39.551+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:39.571 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:39.571 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:39.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:39.651 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:39.652+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:39.682 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:39.682 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:39.691 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:39.752 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:39.753+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:39.792 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:39.793 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:39.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:39.885 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:39.886+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:39.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:21:39.950 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:39.950 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:39.953 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:39.954+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:39.959 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:40.054 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:40.055+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:40.066 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-04-23T21:21:40.105 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.106 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:40.155 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:40.155+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:40.211 INFO:teuthology.orchestra.run.smithi111.stdout:1657 read 0xa2f2ba3 thru 0xa2f60e0 (0x353e bytes) 2024-04-23T21:21:40.212 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.221 INFO:teuthology.orchestra.run.smithi111.stdout:1659 writesame 0xc7cdb28 thru 0xc7dc9b0 data_size 0x73(0xee89 bytes) 2024-04-23T21:21:40.221 INFO:teuthology.orchestra.run.smithi111.stdout:1661 read 0x5342c21 thru 0x534c917 (0x9cf7 bytes) 2024-04-23T21:21:40.231 INFO:teuthology.orchestra.run.smithi111.stdout:1662 trunc from 0xc7dc9b1 to 0x4ec7b26 2024-04-23T21:21:40.256 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:40.257+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:40.264 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.264 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.274 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:40.283 INFO:teuthology.orchestra.run.smithi111.stdout:1663 write 0xaa88765 thru 0xaa8a0be (0x195a bytes) 2024-04-23T21:21:40.286 INFO:teuthology.orchestra.run.smithi111.stdout:1665 write 0x30f7b8f thru 0x3101b00 (0x9f72 bytes) 2024-04-23T21:21:40.327 INFO:teuthology.orchestra.run.smithi111.stdout:1666 write 0x2f6ad90 thru 0x2f7766c (0xc8dd bytes) 2024-04-23T21:21:40.339 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.348 INFO:teuthology.orchestra.run.smithi111.stdout:1669 writesame 0xdac4fbe thru 0xdacaa37 data_size 0x25(0x5a7a bytes) 2024-04-23T21:21:40.348 INFO:teuthology.orchestra.run.smithi111.stdout:1672 trunc from 0xdacaa38 to 0xe09192d 2024-04-23T21:21:40.353 INFO:teuthology.orchestra.run.smithi111.stdout:1673 write 0x108e33c thru 0x109b8a0 (0xd565 bytes) 2024-04-23T21:21:40.357 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:40.357+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:40.392 INFO:teuthology.orchestra.run.smithi111.stdout:1675 punch from 0xb622218 to 0xb62bcd6, (0x9abe bytes) 2024-04-23T21:21:40.392 INFO:teuthology.orchestra.run.smithi111.stdout:1677 write 0x6de0f02 thru 0x6de17ac (0x8ab bytes) 2024-04-23T21:21:40.395 INFO:teuthology.orchestra.run.smithi111.stdout:1678 read 0x49f55cd thru 0x49f738b (0x1dbf bytes) 2024-04-23T21:21:40.396 INFO:teuthology.orchestra.run.smithi111.stdout:1679 read 0x54df4e0 thru 0x54eb032 (0xbb53 bytes) 2024-04-23T21:21:40.405 INFO:teuthology.orchestra.run.smithi111.stdout:1681 trunc from 0xe09192d to 0x593210a 2024-04-23T21:21:40.423 INFO:teuthology.orchestra.run.smithi111.stdout:1682 trunc from 0x593210a to 0xb4d3a01 2024-04-23T21:21:40.433 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.433 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.435 INFO:teuthology.orchestra.run.smithi111.stdout:1683 trunc from 0xb4d3a01 to 0x4da7dc9 2024-04-23T21:21:40.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:40.458 INFO:teuthology.orchestra.run.smithi111.stdout:1684 write 0xdea5e5f thru 0xdea6649 (0x7eb bytes) 2024-04-23T21:21:40.458 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:40.458+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:40.460 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.460 INFO:teuthology.orchestra.run.smithi111.stdout:1688 write 0x87cc614 thru 0x87d546a (0x8e57 bytes) 2024-04-23T21:21:40.468 INFO:teuthology.orchestra.run.smithi111.stdout:1689 punch from 0x5a455de to 0x5a50f1c, (0xb93e bytes) 2024-04-23T21:21:40.477 INFO:teuthology.orchestra.run.smithi111.stdout:1690 trunc from 0xdea664a to 0x700e0c5 2024-04-23T21:21:40.486 INFO:teuthology.orchestra.run.smithi111.stdout:1694 read 0x3af4831 thru 0x3afa638 (0x5e08 bytes) 2024-04-23T21:21:40.486 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.486 INFO:teuthology.orchestra.run.smithi111.stdout:1696 writesame 0x458a7e6 thru 0x458d9b1 data_size 0x2(0x31cc bytes) 2024-04-23T21:21:40.502 INFO:teuthology.orchestra.run.smithi111.stdout:1697 read 0xf9e9fc thru 0xfa1ab3 (0x30b8 bytes) 2024-04-23T21:21:40.503 INFO:teuthology.orchestra.run.smithi111.stdout:1698 write 0x96f498 thru 0x9759b6 (0x651f bytes) 2024-04-23T21:21:40.530 INFO:teuthology.orchestra.run.smithi111.stdout:1701 read 0xfe3c4d thru 0xfe4a64 (0xe18 bytes) 2024-04-23T21:21:40.530 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.536 INFO:teuthology.orchestra.run.smithi111.stdout:1704 writesame 0x780b9f0 thru 0x780d45a data_size 0x1(0x1a6b bytes) 2024-04-23T21:21:40.539 INFO:teuthology.orchestra.run.smithi111.stdout:1705 trunc from 0x780d45b to 0x4f6f87a 2024-04-23T21:21:40.546 INFO:teuthology.orchestra.run.smithi111.stdout:1706 read 0x4bfb3a4 thru 0x4c0a4ac (0xf109 bytes) 2024-04-23T21:21:40.546 INFO:teuthology.orchestra.run.smithi111.stdout:1707 punch from 0x321fbcf to 0x322f6d1, (0xfb02 bytes) 2024-04-23T21:21:40.546 INFO:teuthology.orchestra.run.smithi111.stdout:1708 writesame 0x7a6d39 thru 0x7afdab data_size 0x1(0x9073 bytes) 2024-04-23T21:21:40.558 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:40.559+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:40.563 INFO:teuthology.orchestra.run.smithi111.stdout:1709 write 0x2236a43 thru 0x223ec63 (0x8221 bytes) 2024-04-23T21:21:40.599 INFO:teuthology.orchestra.run.smithi111.stdout:1711 write 0xbfb6733 thru 0xbfbd11e (0x69ec bytes) 2024-04-23T21:21:40.603 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.603 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.605 INFO:teuthology.orchestra.run.smithi111.stdout:1712 punch from 0x9464c0f to 0x946ecfe, (0xa0ef bytes) 2024-04-23T21:21:40.605 INFO:teuthology.orchestra.run.smithi111.stdout:1713 write 0x41771bf thru 0x417c2c7 (0x5109 bytes) 2024-04-23T21:21:40.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:40.617 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.623 INFO:teuthology.orchestra.run.smithi111.stdout:1715 writesame 0xd2473dc thru 0xd25248b data_size 0x58(0xb0b0 bytes) 2024-04-23T21:21:40.630 INFO:teuthology.orchestra.run.smithi111.stdout:1716 write 0xded8b89 thru 0xdee1467 (0x88df bytes) 2024-04-23T21:21:40.635 INFO:teuthology.orchestra.run.smithi111.stdout:1717 punch from 0x9e06c4d to 0x9e1587f, (0xec32 bytes) 2024-04-23T21:21:40.635 INFO:teuthology.orchestra.run.smithi111.stdout:1718 read 0xad24daa thru 0xad2b032 (0x6289 bytes) 2024-04-23T21:21:40.635 INFO:teuthology.orchestra.run.smithi111.stdout:1719 write 0x4f8bac8 thru 0x4f91b16 (0x604f bytes) 2024-04-23T21:21:40.638 INFO:teuthology.orchestra.run.smithi111.stdout:1720 trunc from 0xdee1468 to 0xc02c8a8 2024-04-23T21:21:40.654 INFO:teuthology.orchestra.run.smithi111.stdout:1722 write 0xcaa2145 thru 0xcaa7b17 (0x59d3 bytes) 2024-04-23T21:21:40.659 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:40.659+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:40.659 INFO:teuthology.orchestra.run.smithi111.stdout:1723 punch from 0x5aab7db to 0x5aabd18, (0x53d bytes) 2024-04-23T21:21:40.660 INFO:teuthology.orchestra.run.smithi111.stdout:1724 read 0x1d05b7c thru 0x1d144a6 (0xe92b bytes) 2024-04-23T21:21:40.674 INFO:teuthology.orchestra.run.smithi111.stdout:1725 trunc from 0xcaa7b18 to 0x3bcd14 2024-04-23T21:21:40.731 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.731 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:40.754 INFO:teuthology.orchestra.run.smithi111.stdout:1726 writesame 0x4a0fc3a thru 0x4a193a7 data_size 0x8e(0x976e bytes) 2024-04-23T21:21:40.761 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:40.761+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:40.770 INFO:teuthology.orchestra.run.smithi111.stdout:1727 write 0xc67c3c1 thru 0xc6896da (0xd31a bytes) 2024-04-23T21:21:40.775 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.775 INFO:teuthology.orchestra.run.smithi111.stdout:1729 read 0x48f2040 thru 0x48f38b5 (0x1876 bytes) 2024-04-23T21:21:40.775 INFO:teuthology.orchestra.run.smithi111.stdout:1730 write 0xa1a8408 thru 0xa1af4d8 (0x70d1 bytes) 2024-04-23T21:21:40.781 INFO:teuthology.orchestra.run.smithi111.stdout:1731 trunc from 0xc6896db to 0xa744a69 2024-04-23T21:21:40.791 INFO:teuthology.orchestra.run.smithi111.stdout:1733 writesame 0x1247bb thru 0x12d82b data_size 0x1(0x9071 bytes) 2024-04-23T21:21:40.801 INFO:teuthology.orchestra.run.smithi111.stdout:1734 writesame 0x4e11b1f thru 0x4e1f3fb data_size 0x4d(0xd8dd bytes) 2024-04-23T21:21:40.806 INFO:teuthology.orchestra.run.smithi111.stdout:1738 read 0x80635a4 thru 0x806c6be (0x911b bytes) 2024-04-23T21:21:40.814 INFO:teuthology.orchestra.run.smithi111.stdout:1739 write 0xc522fed thru 0xc53028c (0xd2a0 bytes) 2024-04-23T21:21:40.821 INFO:teuthology.orchestra.run.smithi111.stdout:1740 trunc from 0xc53028d to 0xca8ff5b 2024-04-23T21:21:40.826 INFO:teuthology.orchestra.run.smithi111.stdout:1741 punch from 0xc05b836 to 0xc06826d, (0xca37 bytes) 2024-04-23T21:21:40.826 INFO:teuthology.orchestra.run.smithi111.stdout:1742 read 0x6a8e6be thru 0x6a9d433 (0xed76 bytes) 2024-04-23T21:21:40.826 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.826 INFO:teuthology.orchestra.run.smithi111.stdout:1745 write 0x5a0646c thru 0x5a0fee9 (0x9a7e bytes) 2024-04-23T21:21:40.830 INFO:teuthology.orchestra.run.smithi111.stdout:1749 write 0x800eab2 thru 0x8018a83 (0x9fd2 bytes) 2024-04-23T21:21:40.839 INFO:teuthology.orchestra.run.smithi111.stdout:1752 punch from 0x9b4919e to 0x9b4f772, (0x65d4 bytes) 2024-04-23T21:21:40.839 INFO:teuthology.orchestra.run.smithi111.stdout:1754 read 0x4e2c042 thru 0x4e341de (0x819d bytes) 2024-04-23T21:21:40.839 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.839 INFO:teuthology.orchestra.run.smithi111.stdout:1757 read 0x997c2cb thru 0x9981712 (0x5448 bytes) 2024-04-23T21:21:40.841 INFO:teuthology.orchestra.run.smithi111.stdout:1758 trunc from 0xca8ff5b to 0xe1f9a3c 2024-04-23T21:21:40.846 INFO:teuthology.orchestra.run.smithi111.stdout:1759 writesame 0x271afdb thru 0x27260bc data_size 0x3(0xb0e2 bytes) 2024-04-23T21:21:40.858 INFO:teuthology.orchestra.run.smithi111.stdout:1760 punch from 0x4680189 to 0x468bdcb, (0xbc42 bytes) 2024-04-23T21:21:40.862 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:40.861+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:40.871 INFO:teuthology.orchestra.run.smithi111.stdout:1761 trunc from 0xe1f9a3c to 0x308daaa 2024-04-23T21:21:40.884 INFO:teuthology.orchestra.run.smithi111.stdout:1762 write 0x28e2d14 thru 0x28e5961 (0x2c4e bytes) 2024-04-23T21:21:40.894 INFO:teuthology.orchestra.run.smithi111.stdout:1764 read 0x13b3d46 thru 0x13bae9b (0x7156 bytes) 2024-04-23T21:21:40.894 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:21:40.899 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:40.899 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:40.908 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:40.913 INFO:teuthology.orchestra.run.smithi111.stdout:1765 writesame 0xa0ee6b8 thru 0xa0f1a83 data_size 0x44(0x33cc bytes) 2024-04-23T21:21:40.913 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.913 INFO:teuthology.orchestra.run.smithi111.stdout:1770 read 0x2529296 thru 0x2535c76 (0xc9e1 bytes) 2024-04-23T21:21:40.915 INFO:teuthology.orchestra.run.smithi111.stdout:1771 punch from 0x36709e3 to 0x367d4d7, (0xcaf4 bytes) 2024-04-23T21:21:40.925 INFO:teuthology.orchestra.run.smithi111.stdout:1772 write 0xe2680ea thru 0xe26caa6 (0x49bd bytes) 2024-04-23T21:21:40.931 INFO:teuthology.orchestra.run.smithi111.stdout:1773 read 0x3eebb75 thru 0x3ef92d4 (0xd760 bytes) 2024-04-23T21:21:40.943 INFO:teuthology.orchestra.run.smithi111.stdout:1774 trunc from 0xe26caa7 to 0x48a36e1 2024-04-23T21:21:40.962 INFO:teuthology.orchestra.run.smithi111.stdout:1775 trunc from 0x48a36e1 to 0xc476546 2024-04-23T21:21:40.974 INFO:teuthology.orchestra.run.smithi111.stdout:1777 writesame 0xce42961 thru 0xce51c1d data_size 0x1(0xf2bd bytes) 2024-04-23T21:21:40.974 INFO:teuthology.orchestra.run.smithi111.stdout:1778 read 0x974ff6 thru 0x9775bb (0x25c6 bytes) 2024-04-23T21:21:40.974 INFO:teuthology.orchestra.run.smithi111.stdout:1782 writesame 0xa08c79c thru 0xa08df13 data_size 0x4(0x1778 bytes) 2024-04-23T21:21:40.977 INFO:teuthology.orchestra.run.smithi111.stdout:1784 write 0x613fb3c thru 0x6143f12 (0x43d7 bytes) 2024-04-23T21:21:40.984 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:40.984 INFO:teuthology.orchestra.run.smithi111.stdout:1789 punch from 0x3f965fb to 0x3fa1f9e, (0xb9a3 bytes) 2024-04-23T21:21:40.991 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:40.991+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:40.997 INFO:teuthology.orchestra.run.smithi111.stdout:1790 trunc from 0xce51c1e to 0x1a507ef 2024-04-23T21:21:41.019 INFO:teuthology.orchestra.run.smithi111.stdout:1792 write 0x54463b6 thru 0x54513bc (0xb007 bytes) 2024-04-23T21:21:41.029 INFO:teuthology.orchestra.run.smithi111.stdout:1795 write 0x4140f44 thru 0x4148811 (0x78ce bytes) 2024-04-23T21:21:41.037 INFO:teuthology.orchestra.run.smithi111.stdout:1796 write 0x49d4d3f thru 0x49d82bd (0x357f bytes) 2024-04-23T21:21:41.057 INFO:teuthology.orchestra.run.smithi111.stdout:1797 writesame 0xcec3572 thru 0xcec8e00 data_size 0x3(0x588f bytes) 2024-04-23T21:21:41.057 INFO:teuthology.orchestra.run.smithi111.stdout:1800 write 0x85e01e5 thru 0x85ee5ee (0xe40a bytes) 2024-04-23T21:21:41.059 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:41.059 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:41.063 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:41.063+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:41.063 INFO:teuthology.orchestra.run.smithi111.stdout:1801 punch from 0x97fa666 to 0x980017d, (0x5b17 bytes) 2024-04-23T21:21:41.069 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:41.075 INFO:teuthology.orchestra.run.smithi111.stdout:1802 trunc from 0xcec8e01 to 0x2a6e910 2024-04-23T21:21:41.087 INFO:teuthology.orchestra.run.smithi111.stdout:1803 read 0x129981f thru 0x12a14e4 (0x7cc6 bytes) 2024-04-23T21:21:41.087 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.087 INFO:teuthology.orchestra.run.smithi111.stdout:1805 read 0xb536a6 thru 0xb61d00 (0xe65b bytes) 2024-04-23T21:21:41.096 INFO:teuthology.orchestra.run.smithi111.stdout:1806 trunc from 0x2a6e910 to 0xa3999e0 2024-04-23T21:21:41.102 INFO:teuthology.orchestra.run.smithi111.stdout:1808 trunc from 0xa3999e0 to 0xbd8d346 2024-04-23T21:21:41.107 INFO:teuthology.orchestra.run.smithi111.stdout:1809 punch from 0x9c60632 to 0x9c64b8f, (0x455d bytes) 2024-04-23T21:21:41.108 INFO:teuthology.orchestra.run.smithi111.stdout:1810 writesame 0x4b2ed30 thru 0x4b3258b data_size 0x2(0x385c bytes) 2024-04-23T21:21:41.114 INFO:teuthology.orchestra.run.smithi111.stdout:1812 write 0x8ebdf02 thru 0x8ecba50 (0xdb4f bytes) 2024-04-23T21:21:41.118 INFO:teuthology.orchestra.run.smithi111.stdout:1814 read 0x1780bb5 thru 0x178cbaf (0xbffb bytes) 2024-04-23T21:21:41.118 INFO:teuthology.orchestra.run.smithi111.stdout:1815 write 0x6bee007 thru 0x6bef768 (0x1762 bytes) 2024-04-23T21:21:41.120 INFO:teuthology.orchestra.run.smithi111.stdout:1816 write 0x44bfa4 thru 0x44ce65 (0xec2 bytes) 2024-04-23T21:21:41.122 INFO:teuthology.orchestra.run.smithi111.stdout:1817 writesame 0x5eb7c07 thru 0x5ec38b4 data_size 0x1(0xbcae bytes) 2024-04-23T21:21:41.133 INFO:teuthology.orchestra.run.smithi111.stdout:1818 read 0xb526e01 thru 0xb53397a (0xcb7a bytes) 2024-04-23T21:21:41.142 INFO:teuthology.orchestra.run.smithi111.stdout:1819 write 0xed12de1 thru 0xed1906c (0x628c bytes) 2024-04-23T21:21:41.146 INFO:teuthology.orchestra.run.smithi111.stdout:1820 punch from 0x83d464a to 0x83d49fa, (0x3b0 bytes) 2024-04-23T21:21:41.146 INFO:teuthology.orchestra.run.smithi111.stdout:1821 write 0x7d6999b thru 0x7d6cc41 (0x32a7 bytes) 2024-04-23T21:21:41.150 INFO:teuthology.orchestra.run.smithi111.stdout:1822 read 0x1aa1357 thru 0x1aacc32 (0xb8dc bytes) 2024-04-23T21:21:41.150 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.150 INFO:teuthology.orchestra.run.smithi111.stdout:1825 writesame 0x5800633 thru 0x580c73e data_size 0x23(0xc10c bytes) 2024-04-23T21:21:41.157 INFO:teuthology.orchestra.run.smithi111.stdout:1826 punch from 0xde6354c to 0xde73048, (0xfafc bytes) 2024-04-23T21:21:41.157 INFO:teuthology.orchestra.run.smithi111.stdout:1827 read 0x660e1b5 thru 0x66125ca (0x4416 bytes) 2024-04-23T21:21:41.157 INFO:teuthology.orchestra.run.smithi111.stdout:1829 punch from 0x97453ad to 0x9748533, (0x3186 bytes) 2024-04-23T21:21:41.157 INFO:teuthology.orchestra.run.smithi111.stdout:1830 read 0x95d88a6 thru 0x95dbf03 (0x365e bytes) 2024-04-23T21:21:41.157 INFO:teuthology.orchestra.run.smithi111.stdout:1832 writesame 0xbd41c86 thru 0xbd4dd69 data_size 0x1e(0xc0e4 bytes) 2024-04-23T21:21:41.163 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:41.163+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:41.164 INFO:teuthology.orchestra.run.smithi111.stdout:1833 punch from 0x23b1617 to 0x23c15bc, (0xffa5 bytes) 2024-04-23T21:21:41.164 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.164 INFO:teuthology.orchestra.run.smithi111.stdout:1836 write 0x6c53d37 thru 0x6c557c3 (0x1a8d bytes) 2024-04-23T21:21:41.167 INFO:teuthology.orchestra.run.smithi111.stdout:1838 writesame 0xe97e9db thru 0xe98c94f data_size 0x11(0xdf75 bytes) 2024-04-23T21:21:41.178 INFO:teuthology.orchestra.run.smithi111.stdout:1839 write 0x1639524 thru 0x163b946 (0x2423 bytes) 2024-04-23T21:21:41.184 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:41.184 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:41.191 INFO:teuthology.orchestra.run.smithi111.stdout:1841 trunc from 0xed1906d to 0x77949b0 2024-04-23T21:21:41.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:41.205 INFO:teuthology.orchestra.run.smithi111.stdout:1842 read 0x76244d7 thru 0x762b1f5 (0x6d1f bytes) 2024-04-23T21:21:41.217 INFO:teuthology.orchestra.run.smithi111.stdout:1843 write 0xd7bfbbf thru 0xd7cf34b (0xf78d bytes) 2024-04-23T21:21:41.224 INFO:teuthology.orchestra.run.smithi111.stdout:1844 write 0x19ee16 thru 0x1a6586 (0x7771 bytes) 2024-04-23T21:21:41.225 INFO:teuthology.orchestra.run.smithi111.stdout:1845 writesame 0xb88254d thru 0xb8865d2 data_size 0x3(0x4086 bytes) 2024-04-23T21:21:41.230 INFO:teuthology.orchestra.run.smithi111.stdout:1846 read 0x336eb1f thru 0x337566b (0x6b4d bytes) 2024-04-23T21:21:41.230 INFO:teuthology.orchestra.run.smithi111.stdout:1848 punch from 0x4cb5d68 to 0x4cbbdde, (0x6076 bytes) 2024-04-23T21:21:41.230 INFO:teuthology.orchestra.run.smithi111.stdout:1849 write 0x1d4b5fc thru 0x1d5597f (0xa384 bytes) 2024-04-23T21:21:41.246 INFO:teuthology.orchestra.run.smithi111.stdout:1850 trunc from 0xd7cf34c to 0x48d99be 2024-04-23T21:21:41.264 INFO:teuthology.orchestra.run.smithi111.stdout:1851 read 0x35a08f2 thru 0x35a7735 (0x6e44 bytes) 2024-04-23T21:21:41.264 INFO:teuthology.orchestra.run.smithi111.stdout:1853 writesame 0xc304f8 thru 0xc34820 data_size 0xb(0x4329 bytes) 2024-04-23T21:21:41.265 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:41.264+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:41.268 INFO:teuthology.orchestra.run.smithi111.stdout:1854 writesame 0x37b47e thru 0x37db87 data_size 0x13(0x270a bytes) 2024-04-23T21:21:41.275 INFO:teuthology.orchestra.run.smithi111.stdout:1855 writesame 0x5dd9290 thru 0x5de1138 data_size 0x5(0x7ea9 bytes) 2024-04-23T21:21:41.283 INFO:teuthology.orchestra.run.smithi111.stdout:1856 write 0x768d473 thru 0x769a667 (0xd1f5 bytes) 2024-04-23T21:21:41.292 INFO:teuthology.orchestra.run.smithi111.stdout:1857 punch from 0x59d6775 to 0x59d67e7, (0x72 bytes) 2024-04-23T21:21:41.292 INFO:teuthology.orchestra.run.smithi111.stdout:1859 write 0x21af92f thru 0x21b9385 (0x9a57 bytes) 2024-04-23T21:21:41.300 INFO:teuthology.orchestra.run.smithi111.stdout:1860 writesame 0x347045c thru 0x3475ab9 data_size 0x43(0x565e bytes) 2024-04-23T21:21:41.306 INFO:teuthology.orchestra.run.smithi111.stdout:1862 read 0x1c09861 thru 0x1c0c4be (0x2c5e bytes) 2024-04-23T21:21:41.306 INFO:teuthology.orchestra.run.smithi111.stdout:1863 read 0x2e39a98 thru 0x2e3e0d9 (0x4642 bytes) 2024-04-23T21:21:41.306 INFO:teuthology.orchestra.run.smithi111.stdout:1864 punch from 0x1d385e2 to 0x1d39380, (0xd9e bytes) 2024-04-23T21:21:41.306 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.309 INFO:teuthology.orchestra.run.smithi111.stdout:1866 trunc from 0x769a668 to 0x54cf347 2024-04-23T21:21:41.320 INFO:teuthology.orchestra.run.smithi111.stdout:1867 writesame 0x199cbf4 thru 0x19a3eb6 data_size 0x7(0x72c3 bytes) 2024-04-23T21:21:41.321 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:41.321 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:41.323 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.323 INFO:teuthology.orchestra.run.smithi111.stdout:1874 read 0x18864d0 thru 0x1896034 (0xfb65 bytes) 2024-04-23T21:21:41.340 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:41.340 INFO:teuthology.orchestra.run.smithi111.stdout:1875 write 0xcdba1fb thru 0xcdba69e (0x4a4 bytes) 2024-04-23T21:21:41.343 INFO:teuthology.orchestra.run.smithi111.stdout:1876 writesame 0x69d599d thru 0x69df2ec data_size 0x58(0x9950 bytes) 2024-04-23T21:21:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:1878 read 0x957f2ee thru 0x958f122 (0xfe35 bytes) 2024-04-23T21:21:41.352 INFO:teuthology.orchestra.run.smithi111.stdout:1879 punch from 0x8545668 to 0x854c034, (0x69cc bytes) 2024-04-23T21:21:41.354 INFO:teuthology.orchestra.run.smithi111.stdout:1881 trunc from 0xcdba69f to 0xe8f81fb 2024-04-23T21:21:41.355 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:41.366 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:41.366+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:41.366 INFO:teuthology.orchestra.run.smithi111.stdout:1882 trunc from 0xe8f81fb to 0x894f0d2 2024-04-23T21:21:41.376 INFO:teuthology.orchestra.run.smithi111.stdout:1883 write 0x82d2ef7 thru 0x82e23db (0xf4e5 bytes) 2024-04-23T21:21:41.387 INFO:teuthology.orchestra.run.smithi111.stdout:1884 read 0x26413ed thru 0x2645517 (0x412b bytes) 2024-04-23T21:21:41.399 INFO:teuthology.orchestra.run.smithi111.stdout:1885 writesame 0xdf49483 thru 0xdf4d7a9 data_size 0x1(0x4327 bytes) 2024-04-23T21:21:41.405 INFO:teuthology.orchestra.run.smithi111.stdout:1887 trunc from 0xdf4d7aa to 0x8bed42e 2024-04-23T21:21:41.410 INFO:teuthology.orchestra.run.smithi111.stdout:1888 read 0x259928b thru 0x25a0807 (0x757d bytes) 2024-04-23T21:21:41.410 INFO:teuthology.orchestra.run.smithi111.stdout:1889 read 0x2889c2f thru 0x2893c7a (0xa04c bytes) 2024-04-23T21:21:41.411 INFO:teuthology.orchestra.run.smithi111.stdout:1890 punch from 0x1eb5dcc to 0x1eb928f, (0x34c3 bytes) 2024-04-23T21:21:41.421 INFO:teuthology.orchestra.run.smithi111.stdout:1893 write 0xd0b8e77 thru 0xd0bd94f (0x4ad9 bytes) 2024-04-23T21:21:41.427 INFO:teuthology.orchestra.run.smithi111.stdout:1894 read 0x309d921 thru 0x30a43ce (0x6aae bytes) 2024-04-23T21:21:41.438 INFO:teuthology.orchestra.run.smithi111.stdout:1895 trunc from 0xd0bd950 to 0x403e6cd 2024-04-23T21:21:41.450 INFO:teuthology.orchestra.run.smithi111.stdout:1897 punch from 0x28dece4 to 0x28e5e96, (0x71b2 bytes) 2024-04-23T21:21:41.455 INFO:teuthology.orchestra.run.smithi111.stdout:1900 write 0x45fcb71 thru 0x460a195 (0xd625 bytes) 2024-04-23T21:21:41.461 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:41.462 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:41.467 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:41.466+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:41.469 INFO:teuthology.orchestra.run.smithi111.stdout:1901 writesame 0x160e2f7 thru 0x161456e data_size 0x5c(0x6278 bytes) 2024-04-23T21:21:41.472 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:41.473 INFO:teuthology.orchestra.run.smithi111.stdout:1902 read 0x2fd5c6d thru 0x2fe5559 (0xf8ed bytes) 2024-04-23T21:21:41.473 INFO:teuthology.orchestra.run.smithi111.stdout:1903 read 0x31034d1 thru 0x310bd58 (0x8888 bytes) 2024-04-23T21:21:41.473 INFO:teuthology.orchestra.run.smithi111.stdout:1904 write 0x3393708 thru 0x339e676 (0xaf6f bytes) 2024-04-23T21:21:41.489 INFO:teuthology.orchestra.run.smithi111.stdout:1905 write 0x4a4b8f2 thru 0x4a5076b (0x4e7a bytes) 2024-04-23T21:21:41.492 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:41.492 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:41.492 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:41.495 INFO:teuthology.orchestra.run.smithi111.stdout:1906 read 0x342994e thru 0x342b017 (0x16ca bytes) 2024-04-23T21:21:41.496 INFO:teuthology.orchestra.run.smithi111.stdout:1908 punch from 0x39fb0f6 to 0x3a0253a, (0x7444 bytes) 2024-04-23T21:21:41.496 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.496 INFO:teuthology.orchestra.run.smithi111.stdout:1910 read 0x324e71e thru 0x3256acd (0x83b0 bytes) 2024-04-23T21:21:41.506 INFO:teuthology.orchestra.run.smithi111.stdout:1911 trunc from 0x4a5076c to 0xd52ba1a 2024-04-23T21:21:41.511 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:41.511 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:41.527 INFO:teuthology.orchestra.run.smithi111.stdout:1913 trunc from 0xd52ba1a to 0x1054685 2024-04-23T21:21:41.529 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:41.529 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:41.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:41.550 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:41.553 INFO:teuthology.orchestra.run.smithi111.stdout:1914 trunc from 0x1054685 to 0xb99707c 2024-04-23T21:21:41.560 INFO:teuthology.orchestra.run.smithi111.stdout:1915 read 0x5142b00 thru 0x5151451 (0xe952 bytes) 2024-04-23T21:21:41.561 INFO:teuthology.orchestra.run.smithi111.stdout:1916 trunc from 0xb99707c to 0xa69dd9a 2024-04-23T21:21:41.567 INFO:teuthology.orchestra.run.smithi111.stdout:1917 writesame 0x17bbd69 thru 0x17c344d data_size 0x1(0x76e5 bytes) 2024-04-23T21:21:41.567 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:41.567+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:41.569 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:41.570 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:41.587 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:41.587 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:41.590 INFO:teuthology.orchestra.run.smithi111.stdout:1918 writesame 0xcd91205 thru 0xcd93129 data_size 0x7(0x1f25 bytes) 2024-04-23T21:21:41.590 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.590 INFO:teuthology.orchestra.run.smithi111.stdout:1921 punch from 0x6119feb to 0x6125b54, (0xbb69 bytes) 2024-04-23T21:21:41.591 INFO:teuthology.orchestra.run.smithi111.stdout:1922 write 0x5611ea7 thru 0x561b897 (0x99f1 bytes) 2024-04-23T21:21:41.591 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:41.591 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:41.591 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:41.591 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:41.596 INFO:teuthology.orchestra.run.smithi111.stdout:1924 read 0x5483998 thru 0x5489d73 (0x63dc bytes) 2024-04-23T21:21:41.597 INFO:teuthology.orchestra.run.smithi111.stdout:1925 read 0xc5291d7 thru 0xc533b3c (0xa966 bytes) 2024-04-23T21:21:41.597 INFO:teuthology.orchestra.run.smithi111.stdout:1926 read 0x281ff12 thru 0x282d9f6 (0xdae5 bytes) 2024-04-23T21:21:41.597 INFO:teuthology.orchestra.run.smithi111.stdout:1927 write 0x508bae2 thru 0x508e7d7 (0x2cf6 bytes) 2024-04-23T21:21:41.599 INFO:teuthology.orchestra.run.smithi111.stdout:1928 punch from 0x27019f0 to 0x2709f1b, (0x852b bytes) 2024-04-23T21:21:41.599 INFO:teuthology.orchestra.run.smithi111.stdout:1929 read 0x9267679 thru 0x926a59d (0x2f25 bytes) 2024-04-23T21:21:41.599 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.599 INFO:teuthology.orchestra.run.smithi111.stdout:1931 write 0x22cd7c2 thru 0x22d2236 (0x4a75 bytes) 2024-04-23T21:21:41.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:41.606 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:41.607 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:41.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:41.614 INFO:teuthology.orchestra.run.smithi111.stdout:1932 trunc from 0xcd9312a to 0x358c4d6 2024-04-23T21:21:41.623 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.632 INFO:teuthology.orchestra.run.smithi111.stdout:1934 write 0x5968284 thru 0x5975961 (0xd6de bytes) 2024-04-23T21:21:41.640 INFO:teuthology.orchestra.run.smithi111.stdout:1937 trunc from 0x5975962 to 0x55c57a3 2024-04-23T21:21:41.651 INFO:teuthology.orchestra.run.smithi111.stdout:1938 writesame 0x25513f thru 0x258545 data_size 0x1(0x3407 bytes) 2024-04-23T21:21:41.657 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.662 INFO:teuthology.orchestra.run.smithi111.stdout:1940 trunc from 0x55c57a3 to 0x944835f 2024-04-23T21:21:41.668 INFO:teuthology.orchestra.run.smithi111.stdout:1941 read 0x20960fb thru 0x209d47a (0x7380 bytes) 2024-04-23T21:21:41.668 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.668 INFO:teuthology.orchestra.run.smithi111.stdout:1946 writesame 0x7cad88d thru 0x7cadef3 data_size 0x1(0x667 bytes) 2024-04-23T21:21:41.668 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:41.667+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:41.671 INFO:teuthology.orchestra.run.smithi111.stdout:1947 write 0x5f0b580 thru 0x5f1867c (0xd0fd bytes) 2024-04-23T21:21:41.678 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.686 INFO:teuthology.orchestra.run.smithi111.stdout:1950 write 0xb2e3f93 thru 0xb2ea50b (0x6579 bytes) 2024-04-23T21:21:41.695 INFO:teuthology.orchestra.run.smithi111.stdout:1952 write 0x4432ace thru 0x443c482 (0x99b5 bytes) 2024-04-23T21:21:41.705 INFO:teuthology.orchestra.run.smithi111.stdout:1953 read 0x441f89c thru 0x441febb (0x620 bytes) 2024-04-23T21:21:41.705 INFO:teuthology.orchestra.run.smithi111.stdout:1954 read 0x496d389 thru 0x4971f2e (0x4ba6 bytes) 2024-04-23T21:21:41.705 INFO:teuthology.orchestra.run.smithi111.stdout:1955 writesame 0xcc71b5 thru 0xcca20d data_size 0x1(0x3059 bytes) 2024-04-23T21:21:41.712 INFO:teuthology.orchestra.run.smithi111.stdout:1958 write 0x66847d9 thru 0x668e327 (0x9b4f bytes) 2024-04-23T21:21:41.720 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:41.720 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:41.720 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:41.720 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:41.720 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:41.721 INFO:teuthology.orchestra.run.smithi111.stdout:1959 read 0x5526f14 thru 0x552a255 (0x3342 bytes) 2024-04-23T21:21:41.721 INFO:teuthology.orchestra.run.smithi111.stdout:1960 read 0x31c5e22 thru 0x31ca765 (0x4944 bytes) 2024-04-23T21:21:41.721 INFO:teuthology.orchestra.run.smithi111.stdout:1961 write 0x88939ea thru 0x88a038e (0xc9a5 bytes) 2024-04-23T21:21:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:1963 punch from 0x8e6b5c5 to 0x8e70990, (0x53cb bytes) 2024-04-23T21:21:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.727 INFO:teuthology.orchestra.run.smithi111.stdout:1965 write 0x8964cec thru 0x897004f (0xb364 bytes) 2024-04-23T21:21:41.729 INFO:teuthology.orchestra.run.smithi111.stdout:1969 read 0x567225 thru 0x56cc32 (0x5a0e bytes) 2024-04-23T21:21:41.729 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.729 INFO:teuthology.orchestra.run.smithi111.stdout:1971 read 0x916a261 thru 0x916a486 (0x226 bytes) 2024-04-23T21:21:41.729 INFO:teuthology.orchestra.run.smithi111.stdout:1973 read 0x3a0fc5 thru 0x3adccf (0xcd0b bytes) 2024-04-23T21:21:41.730 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:41.731 INFO:teuthology.orchestra.run.smithi111.stdout:1974 read 0x4ac88a2 thru 0x4acb1df (0x293e bytes) 2024-04-23T21:21:41.731 INFO:teuthology.orchestra.run.smithi111.stdout:1975 read 0x11c0a05 thru 0x11c4f09 (0x4505 bytes) 2024-04-23T21:21:41.731 INFO:teuthology.orchestra.run.smithi111.stdout:1976 write 0x2587c7c thru 0x2588fb1 (0x1336 bytes) 2024-04-23T21:21:41.739 INFO:teuthology.orchestra.run.smithi111.stdout:1977 write 0xc029b2e thru 0xc02f2f2 (0x57c5 bytes) 2024-04-23T21:21:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.742 INFO:teuthology.orchestra.run.smithi111.stdout:1980 write 0x8c6d5f7 thru 0x8c74368 (0x6d72 bytes) 2024-04-23T21:21:41.747 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.747 INFO:teuthology.orchestra.run.smithi111.stdout:1982 punch from 0xa0aa62a to 0xa0b8c29, (0xe5ff bytes) 2024-04-23T21:21:41.753 INFO:teuthology.orchestra.run.smithi111.stdout:1983 write 0xcac3bbb thru 0xcacb04d (0x7493 bytes) 2024-04-23T21:21:41.768 INFO:teuthology.orchestra.run.smithi111.stdout:1985 trunc from 0xcacb04e to 0x3f959d7 2024-04-23T21:21:41.769 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:41.768+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:41.789 INFO:teuthology.orchestra.run.smithi111.stdout:1987 read 0x3c59b2b thru 0x3c5b9c5 (0x1e9b bytes) 2024-04-23T21:21:41.789 INFO:teuthology.orchestra.run.smithi111.stdout:1988 read 0x44e425 thru 0x44e85c (0x438 bytes) 2024-04-23T21:21:41.797 INFO:teuthology.orchestra.run.smithi111.stdout:1989 trunc from 0x3f959d7 to 0xab8999b 2024-04-23T21:21:41.802 INFO:teuthology.orchestra.run.smithi111.stdout:1990 punch from 0x8056fef to 0x8064b92, (0xdba3 bytes) 2024-04-23T21:21:41.809 INFO:teuthology.orchestra.run.smithi111.stdout:1991 write 0xc4b5511 thru 0xc4b7dae (0x289e bytes) 2024-04-23T21:21:41.812 INFO:teuthology.orchestra.run.smithi111.stdout:1993 write 0x45c00cc thru 0x45ca49f (0xa3d4 bytes) 2024-04-23T21:21:41.821 INFO:teuthology.orchestra.run.smithi111.stdout:1995 write 0x83acc90 thru 0x83b777d (0xaaee bytes) 2024-04-23T21:21:41.828 INFO:teuthology.orchestra.run.smithi111.stdout:1996 writesame 0x6b05ef7 thru 0x6b0d505 data_size 0x1(0x760f bytes) 2024-04-23T21:21:41.842 INFO:teuthology.orchestra.run.smithi111.stdout:1997 punch from 0x46909e0 to 0x4692ef3, (0x2513 bytes) 2024-04-23T21:21:41.843 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.843 INFO:teuthology.orchestra.run.smithi111.stdout:1999 punch from 0x10b627a to 0x10bd6f2, (0x7478 bytes) 2024-04-23T21:21:41.843 INFO:teuthology.orchestra.run.smithi111.stdout:2000 punch from 0x7b3114 to 0x7bda7d, (0xa969 bytes) 2024-04-23T21:21:41.843 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.843 INFO:teuthology.orchestra.run.smithi111.stdout:2003 writesame 0x2f39425 thru 0x2f45554 data_size 0x58(0xc130 bytes) 2024-04-23T21:21:41.852 INFO:teuthology.orchestra.run.smithi111.stdout:2004 write 0xcb22ed thru 0xcbd38e (0xb0a2 bytes) 2024-04-23T21:21:41.856 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:7796 3.9 e27/18 3:9154b378:::rbd_data.111830ac55a1.0000000000000028:head v 27'28, mlcod=27'26)", 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 11.863533837, 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046354710000000004, 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:41.857 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993118+0000", 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.68e-07 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993123+0000", 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993133+0000", 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0240000000000003e-06 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993137+0000", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993191+0000", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3300000000000001e-05 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.993213+0000", 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.23e-05 2024-04-23T21:21:41.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.997710+0000", 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0044970469999999997 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.997753+0000", 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3373e-05 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:10838 3.9 e31/18 3:9387dc3a:::rbd_data.111823663ad5.0000000000000001:head v 31'44, mlcod=30'42)", 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.6131800570000001, 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049523199999999996, 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243472+0000", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.243473+0000", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3459999999999999e-06 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244449+0000", 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00097653599999999996 2024-04-23T21:21:41.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244463+0000", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3699e-05 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244471+0000", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9810000000000003e-06 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244528+0000", 2024-04-23T21:21:41.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7246e-05 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.244551+0000", 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2331999999999999e-05 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.248369+0000", 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0038185010000000002 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:34.248424+0000", 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4679000000000003e-05 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:13154 3.0 3:0277b7fa:::rbd_data.11188429226.000000000000000d:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.5070456810000001, 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0062056239999999999, 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:41.866 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 13154 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349606+0000", 2024-04-23T21:21:41.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349608+0000", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3539999999999998e-06 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349617+0000", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6959999999999994e-06 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349636+0000", 2024-04-23T21:21:41.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9392e-05 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349647+0000", 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0199e-05 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349701+0000", 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4910000000000001e-05 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.349757+0000", 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5605e-05 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355661+0000", 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005903686 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355723+0000", 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2292000000000006e-05 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:37.355812+0000", 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8490000000000001e-05 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14425 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 3342336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5522812259999998, 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004649355, 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.871 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14425 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304370+0000", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304371+0000", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6100000000000001e-07 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304373+0000", 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3120000000000001e-06 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304380+0000", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.917e-06 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304385+0000", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1680000000000003e-06 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.308891+0000", 2024-04-23T21:21:41.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0045060159999999998 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.308941+0000", 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9360000000000002e-05 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309020+0000", 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9021000000000001e-05 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14426 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 3866624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5522596269999998, 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004767777, 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14426 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.876 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304392+0000", 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3799999999999998e-07 2024-04-23T21:21:41.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304394+0000", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0020000000000001e-06 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304402+0000", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8690000000000004e-06 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304407+0000", 2024-04-23T21:21:41.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0930000000000002e-06 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309045+0000", 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046372829999999999 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309088+0000", 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3841000000000003e-05 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309160+0000", 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1250999999999999e-05 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14427 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2293760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5522376959999997, 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004892179, 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.880 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14427 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304414+0000", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1900000000000003e-07 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304416+0000", 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9819999999999998e-06 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304430+0000", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.358e-05 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.304444+0000", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3599e-05 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309185+0000", 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047414550000000003 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309231+0000", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5896999999999998e-05 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.309306+0000", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5147000000000004e-05 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14493 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 4128768~25853] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:41.884 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5508752220000002, 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047371679999999999, 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14493 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:41.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305776+0000", 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305777+0000", 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305781+0000", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.856e-06 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305791+0000", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5060000000000001e-06 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305797+0000", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5139999999999998e-06 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310152+0000", 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0043545600000000004 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310194+0000", 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2179999999999999e-05 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310514+0000", 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031975299999999999 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14494 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2555904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5508449510000002, 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048527910000000004, 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14494 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305807+0000", 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3200000000000005e-07 2024-04-23T21:21:41.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305809+0000", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2170000000000001e-06 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305817+0000", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2320000000000004e-06 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305822+0000", 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4e-06 2024-04-23T21:21:41.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310542+0000", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047198810000000004 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310586+0000", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4011999999999998e-05 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310659+0000", 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3417000000000005e-05 2024-04-23T21:21:41.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14495 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 3080192~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5508228509999999, 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049765809999999999, 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.893 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14495 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305829+0000", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0200000000000003e-07 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305831+0000", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305839+0000", 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305844+0000", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1730000000000003e-06 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310684+0000", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048400689999999998 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310728+0000", 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4002000000000003e-05 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310805+0000", 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7349999999999996e-05 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14496 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 3604480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5508010900000002, 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050903179999999999, 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.897 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14496 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:41.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305851+0000", 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4799999999999998e-07 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305853+0000", 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305859+0000", 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2330000000000003e-06 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305872+0000", 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2296999999999999e-05 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310828+0000", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.004956645 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310869+0000", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0229999999999999e-05 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310941+0000", 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2296e-05 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.901 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14497 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.5507588839999999, 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0051725909999999998, 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14497 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305893+0000", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305896+0000", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8590000000000001e-06 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305904+0000", 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5340000000000002e-06 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.305915+0000", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.113e-05 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310955+0000", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005040615 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.310992+0000", 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6785000000000001e-05 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.311065+0000", 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3057000000000002e-05 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14965 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1441792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.3623772750000001, 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046185159999999996, 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.906 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14965 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494274+0000", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494275+0000", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8700000000000001e-07 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494277+0000", 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0490000000000002e-06 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494283+0000", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494287+0000", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8099999999999997e-06 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498665+0000", 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0043777579999999998 2024-04-23T21:21:41.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498747+0000", 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1791000000000003e-05 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498893+0000", 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000145878 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.910 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:14966 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1966080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.362358371, 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0047961180000000003, 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 14966 2024-04-23T21:21:41.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494293+0000", 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494294+0000", 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494295+0000", 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8029999999999999e-06 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494301+0000", 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6869999999999999e-06 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494306+0000", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5639999999999998e-06 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498930+0000", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0046239940000000002 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.498978+0000", 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8266999999999997e-05 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499089+0000", 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011142299999999999 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15024 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.3619173469999999, 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048110970000000003, 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.915 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15024 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494734+0000", 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494735+0000", 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4400000000000001e-07 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494737+0000", 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494744+0000", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0890000000000002e-06 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494749+0000", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2560000000000002e-06 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499335+0000", 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0045854670000000002 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499415+0000", 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0197000000000005e-05 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499545+0000", 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013021300000000001 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15025 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 1507328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.919 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.361894682, 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050516770000000001, 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15025 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494757+0000", 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.65e-07 2024-04-23T21:21:41.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494759+0000", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9850000000000001e-06 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494767+0000", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2179999999999999e-06 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494773+0000", 2024-04-23T21:21:41.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7849999999999997e-06 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499588+0000", 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0048144540000000001 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499677+0000", 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9622000000000004e-05 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499809+0000", 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013124799999999999 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15026 3.4 3:20a191d5:::rbd_data.11188429226.000000000000000a:e [sparse-read 2031616~52822] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.3618715039999998, 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005573031, 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.924 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15026 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494780+0000", 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494781+0000", 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494795+0000", 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4107999999999999e-05 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494802+0000", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6229999999999997e-06 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494816+0000", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3553999999999999e-05 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499863+0000", 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0050470280000000003 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.499961+0000", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8034000000000004e-05 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500353+0000", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000392273 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15027 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 458752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:41.928 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.3618281539999999, 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0058122069999999998, 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15027 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:41.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494823+0000", 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494824+0000", 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5600000000000001e-07 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494826+0000", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2010000000000002e-06 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494834+0000", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2199999999999992e-06 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494839+0000", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.011e-06 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500400+0000", 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0055606379999999997 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500495+0000", 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4664e-05 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500636+0000", 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014101700000000001 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15028 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 983040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.361805999, 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0059530670000000003, 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15028 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494846+0000", 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494848+0000", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9010000000000001e-06 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494855+0000", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6030000000000001e-06 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494859+0000", 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.831e-06 2024-04-23T21:21:41.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500667+0000", 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0058079400000000002 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500710+0000", 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2803000000000003e-05 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500799+0000", 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8438999999999998e-05 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15029 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 1507328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.361785829, 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.006189295, 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.937 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15029 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494866+0000", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494870+0000", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5319999999999998e-06 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494877+0000", 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7479999999999993e-06 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494883+0000", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3029999999999999e-06 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500836+0000", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0059528910000000001 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.500917+0000", 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.14e-05 2024-04-23T21:21:41.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501055+0000", 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013802199999999999 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:15030 3.4 3:2155fc18:::rbd_data.11188429226.000000000000000b:e [sparse-read 2031616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e35)", 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 3.3617615499999998, 2024-04-23T21:21:41.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0063112130000000004, 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 15030 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:41.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494890+0000", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494891+0000", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1600000000000001e-07 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494893+0000", 2024-04-23T21:21:41.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0880000000000002e-06 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494905+0000", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1875999999999999e-05 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.494917+0000", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2024999999999999e-05 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501074+0000", 2024-04-23T21:21:41.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0061574120000000001 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501127+0000", 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2602000000000002e-05 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:38.501201+0000", 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4593999999999998e-05 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:41.945 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:41.946 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:41.946 INFO:teuthology.orchestra.run.smithi111.stdout:2005 punch from 0x3ec6af8 to 0x3ed035f, (0x9867 bytes) 2024-04-23T21:21:41.946 INFO:teuthology.orchestra.run.smithi111.stdout:2006 trunc from 0xc4b7daf to 0x1afe8ce 2024-04-23T21:21:41.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:41.948 INFO:teuthology.orchestra.run.smithi111.stdout:2008 read 0x39ab61 thru 0x3a0c16 (0x60b6 bytes) 2024-04-23T21:21:41.948 INFO:teuthology.orchestra.run.smithi111.stdout:2009 read 0x106b131 thru 0x106f927 (0x47f7 bytes) 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2010 write 0x7afe8cc thru 0x7b0bd53 (0xd488 bytes) 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2011 read 0x672062e thru 0x6723808 (0x31db bytes) 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2012 write 0x3f99f1d thru 0x3f9ac0f (0xcf3 bytes) 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2014 writesame 0x7513523 thru 0x751b800 data_size 0x2(0x82de bytes) 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2015 trunc from 0x7b0bd54 to 0x92b78c3 2024-04-23T21:21:41.949 INFO:teuthology.orchestra.run.smithi111.stdout:2017 trunc from 0x92b78c3 to 0x597f455 2024-04-23T21:21:41.950 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:41.950+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:41.954 INFO:teuthology.orchestra.run.smithi111.stdout:2018 write 0xe07e353 thru 0xe0872d8 (0x8f86 bytes) 2024-04-23T21:21:41.960 INFO:teuthology.orchestra.run.smithi111.stdout:2019 writesame 0x11b421 thru 0x11e381 data_size 0xd(0x2f61 bytes) 2024-04-23T21:21:41.962 INFO:teuthology.orchestra.run.smithi111.stdout:2020 read 0x2411507 thru 0x2411dbe (0x8b8 bytes) 2024-04-23T21:21:41.962 INFO:teuthology.orchestra.run.smithi111.stdout:2021 write 0x4c7e05b thru 0x4c80af7 (0x2a9d bytes) 2024-04-23T21:21:41.967 INFO:teuthology.orchestra.run.smithi111.stdout:2022 writesame 0x6f49bfd thru 0x6f54881 data_size 0x49(0xac85 bytes) 2024-04-23T21:21:41.972 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:41.972 INFO:teuthology.orchestra.run.smithi111.stdout:2025 writesame 0xa02229a thru 0xa0280d3 data_size 0x7(0x5e3a bytes) 2024-04-23T21:21:41.993 INFO:teuthology.orchestra.run.smithi111.stdout:2028 trunc from 0xe0872d9 to 0xb5232b 2024-04-23T21:21:42.008 INFO:teuthology.orchestra.run.smithi111.stdout:2029 punch from 0x55c6eb to 0x563bf0, (0x7505 bytes) 2024-04-23T21:21:42.008 INFO:teuthology.orchestra.run.smithi111.stdout:2030 read 0x86458a thru 0x86cde0 (0x8857 bytes) 2024-04-23T21:21:42.020 INFO:teuthology.orchestra.run.smithi111.stdout:2031 write 0x64d88d9 thru 0x64d8d85 (0x4ad bytes) 2024-04-23T21:21:42.024 INFO:teuthology.orchestra.run.smithi111.stdout:2032 punch from 0x52919b9 to 0x529961e, (0x7c65 bytes) 2024-04-23T21:21:42.024 INFO:teuthology.orchestra.run.smithi111.stdout:2035 write 0x36eed19 thru 0x36fcffe (0xe2e6 bytes) 2024-04-23T21:21:42.031 INFO:teuthology.orchestra.run.smithi111.stdout:2036 write 0x1f2d6c2 thru 0x1f3c21f (0xeb5e bytes) 2024-04-23T21:21:42.043 INFO:teuthology.orchestra.run.smithi111.stdout:2039 trunc from 0x64d8d86 to 0x2beca7c 2024-04-23T21:21:42.050 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:42.050+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:42.052 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.052 INFO:teuthology.orchestra.run.smithi111.stdout:2041 writesame 0x12ee612 thru 0x12fc9d0 data_size 0x1(0xe3bf bytes) 2024-04-23T21:21:42.067 INFO:teuthology.orchestra.run.smithi111.stdout:2042 writesame 0xd3fbf8 thru 0xd4f8d6 data_size 0x6b(0xfcdf bytes) 2024-04-23T21:21:42.068 INFO:teuthology.orchestra.run.smithi111.stdout:2043 punch from 0x210b5e5 to 0x210c1ab, (0xbc6 bytes) 2024-04-23T21:21:42.069 INFO:teuthology.orchestra.run.smithi111.stdout:2044 read 0x18721d thru 0x18f6c3 (0x84a7 bytes) 2024-04-23T21:21:42.069 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.069 INFO:teuthology.orchestra.run.smithi111.stdout:2046 punch from 0xa90f5f to 0xa999f4, (0x8a95 bytes) 2024-04-23T21:21:42.071 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:42.071 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:42.071 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:42.071 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:42.080 INFO:teuthology.orchestra.run.smithi111.stdout:2047 trunc from 0x2beca7c to 0xbf97bc4 2024-04-23T21:21:42.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:42.086 INFO:teuthology.orchestra.run.smithi111.stdout:2050 write 0x7c6aac1 thru 0x7c6b54a (0xa8a bytes) 2024-04-23T21:21:42.088 INFO:teuthology.orchestra.run.smithi111.stdout:2051 read 0xb5fab6c thru 0xb604e80 (0xa315 bytes) 2024-04-23T21:21:42.088 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.089 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.101 INFO:teuthology.orchestra.run.smithi111.stdout:2054 trunc from 0xbf97bc4 to 0x102a4e0 2024-04-23T21:21:42.114 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.115 INFO:teuthology.orchestra.run.smithi111.stdout:2056 read 0x4c36ad thru 0x4d2525 (0xee79 bytes) 2024-04-23T21:21:42.129 INFO:teuthology.orchestra.run.smithi111.stdout:2058 write 0x83650d7 thru 0x8367ca1 (0x2bcb bytes) 2024-04-23T21:21:42.132 INFO:teuthology.orchestra.run.smithi111.stdout:2059 write 0x3e3be63 thru 0x3e43d17 (0x7eb5 bytes) 2024-04-23T21:21:42.136 INFO:teuthology.orchestra.run.smithi111.stdout:2060 trunc from 0x8367ca2 to 0x832f50b 2024-04-23T21:21:42.142 INFO:teuthology.orchestra.run.smithi111.stdout:2062 punch from 0x62d0e13 to 0x62d4aec, (0x3cd9 bytes) 2024-04-23T21:21:42.147 INFO:teuthology.orchestra.run.smithi111.stdout:2063 trunc from 0x832f50b to 0xb947ed1 2024-04-23T21:21:42.151 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:42.151+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:42.152 INFO:teuthology.orchestra.run.smithi111.stdout:2065 read 0xb58f51b thru 0xb5951c6 (0x5cac bytes) 2024-04-23T21:21:42.153 INFO:teuthology.orchestra.run.smithi111.stdout:2066 read 0x98f304d thru 0x9902bbb (0xfb6f bytes) 2024-04-23T21:21:42.153 INFO:teuthology.orchestra.run.smithi111.stdout:2067 write 0x46b78f2 thru 0x46bc325 (0x4a34 bytes) 2024-04-23T21:21:42.162 INFO:teuthology.orchestra.run.smithi111.stdout:2068 read 0x88f56c2 thru 0x88f8cb0 (0x35ef bytes) 2024-04-23T21:21:42.170 INFO:teuthology.orchestra.run.smithi111.stdout:2069 writesame 0xbfd9208 thru 0xbfde2ae data_size 0x1(0x50a7 bytes) 2024-04-23T21:21:42.170 INFO:teuthology.orchestra.run.smithi111.stdout:2071 write 0x39ac932 thru 0x39bb43e (0xeb0d bytes) 2024-04-23T21:21:42.182 INFO:teuthology.orchestra.run.smithi111.stdout:2072 writesame 0x91f85cc thru 0x91ff811 data_size 0x1(0x7246 bytes) 2024-04-23T21:21:42.192 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.192 INFO:teuthology.orchestra.run.smithi111.stdout:2074 read 0xb64213b thru 0xb64d49e (0xb364 bytes) 2024-04-23T21:21:42.192 INFO:teuthology.orchestra.run.smithi111.stdout:2075 read 0x20f67ae thru 0x2100d14 (0xa567 bytes) 2024-04-23T21:21:42.192 INFO:teuthology.orchestra.run.smithi111.stdout:2077 read 0x3b2f51c thru 0x3b397ed (0xa2d2 bytes) 2024-04-23T21:21:42.193 INFO:teuthology.orchestra.run.smithi111.stdout:2079 read 0x10f013a thru 0x10f6f16 (0x6ddd bytes) 2024-04-23T21:21:42.193 INFO:teuthology.orchestra.run.smithi111.stdout:2081 punch from 0x1182d2 to 0x121170, (0x8e9e bytes) 2024-04-23T21:21:42.193 INFO:teuthology.orchestra.run.smithi111.stdout:2082 write 0x9565975 thru 0x9565bfc (0x288 bytes) 2024-04-23T21:21:42.197 INFO:teuthology.orchestra.run.smithi111.stdout:2083 write 0x6273eb1 thru 0x62839fd (0xfb4d bytes) 2024-04-23T21:21:42.201 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:42.201 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:42.201 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:42.201 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:42.201 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:42.203 INFO:teuthology.orchestra.run.smithi111.stdout:2085 read 0xad02a8f thru 0xad11f9f (0xf511 bytes) 2024-04-23T21:21:42.203 INFO:teuthology.orchestra.run.smithi111.stdout:2088 writesame 0x2af436 thru 0x2bce3a data_size 0x1(0xda05 bytes) 2024-04-23T21:21:42.211 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:42.229 INFO:teuthology.orchestra.run.smithi111.stdout:2089 trunc from 0xbfde2af to 0x16478e6 2024-04-23T21:21:42.243 INFO:teuthology.orchestra.run.smithi111.stdout:2090 trunc from 0x16478e6 to 0x2c71b 2024-04-23T21:21:42.252 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:42.252+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:42.259 INFO:teuthology.orchestra.run.smithi111.stdout:2092 read 0x1ac8 thru 0x10e07 (0xf340 bytes) 2024-04-23T21:21:42.282 INFO:teuthology.orchestra.run.smithi111.stdout:2093 writesame 0xdd8c8d4 thru 0xdd97a6f data_size 0x24(0xb19c bytes) 2024-04-23T21:21:42.282 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.282 INFO:teuthology.orchestra.run.smithi111.stdout:2095 read 0x43a1fe9 thru 0x43ab67f (0x9697 bytes) 2024-04-23T21:21:42.284 INFO:teuthology.orchestra.run.smithi111.stdout:2096 trunc from 0xdd97a70 to 0xc3193ec 2024-04-23T21:21:42.296 INFO:teuthology.orchestra.run.smithi111.stdout:2097 trunc from 0xc3193ec to 0x61447d7 2024-04-23T21:21:42.302 INFO:teuthology.orchestra.run.smithi111.stdout:2098 trunc from 0x61447d7 to 0x5137170 2024-04-23T21:21:42.308 INFO:teuthology.orchestra.run.smithi111.stdout:2099 read 0x3518841 thru 0x3526520 (0xdce0 bytes) 2024-04-23T21:21:42.317 INFO:teuthology.orchestra.run.smithi111.stdout:2100 writesame 0x849b066 thru 0x84aa437 data_size 0x67(0xf3d2 bytes) 2024-04-23T21:21:42.317 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.317 INFO:teuthology.orchestra.run.smithi111.stdout:2102 write 0xbec984 thru 0xbfba3f (0xf0bc bytes) 2024-04-23T21:21:42.327 INFO:teuthology.orchestra.run.smithi111.stdout:2104 punch from 0x523c156 to 0x524bf88, (0xfe32 bytes) 2024-04-23T21:21:42.328 INFO:teuthology.orchestra.run.smithi111.stdout:2105 read 0x13cac92 thru 0x13cccbd (0x202c bytes) 2024-04-23T21:21:42.328 INFO:teuthology.orchestra.run.smithi111.stdout:2106 read 0xd1e616 thru 0xd27da0 (0x978b bytes) 2024-04-23T21:21:42.329 INFO:teuthology.orchestra.run.smithi111.stdout:2108 trunc from 0x84aa438 to 0x7217022 2024-04-23T21:21:42.338 INFO:teuthology.orchestra.run.smithi111.stdout:2111 trunc from 0x7217022 to 0xaf9dc97 2024-04-23T21:21:42.350 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7260 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6684672~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.898185664, 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026061528, 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7260 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451671+0000", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451672+0000", 2024-04-23T21:21:42.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3179999999999999e-06 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451676+0000", 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4199999999999999e-06 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451680+0000", 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6279999999999998e-06 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477564+0000", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025884564999999998 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477641+0000", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7300999999999997e-05 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477732+0000", 2024-04-23T21:21:42.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0884999999999995e-05 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7261 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6881280~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.898171494, 2024-04-23T21:21:42.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026239373, 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7261 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:42.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451685+0000", 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5100000000000001e-07 2024-04-23T21:21:42.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451688+0000", 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.255e-06 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451693+0000", 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8540000000000002e-06 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451698+0000", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4610000000000001e-06 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477761+0000", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026062980999999999 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477836+0000", 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5496999999999999e-05 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477924+0000", 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7973999999999995e-05 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7262 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7077888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.898153589, 2024-04-23T21:21:42.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026422265, 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7262 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:42.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451703+0000", 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9200000000000002e-07 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451705+0000", 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8250000000000001e-06 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451712+0000", 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.759e-06 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451716+0000", 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2889999999999999e-06 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.477952+0000", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026235967999999998 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478032+0000", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0296999999999994e-05 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478125+0000", 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2734999999999999e-05 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.365 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7263 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7798784~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.898132857, 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026605936, 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7263 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.451724+0000", 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452459+0000", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00073525200000000004 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452464+0000", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8940000000000001e-06 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452474+0000", 2024-04-23T21:21:42.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0454e-05 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452481+0000", 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7469999999999999e-06 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478153+0000", 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025672186 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478236+0000", 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3061999999999999e-05 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478329+0000", 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3341e-05 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7265 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8192000~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.89734294, 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026201490000000001, 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7265 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452513+0000", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452514+0000", 2024-04-23T21:21:42.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7899999999999999e-07 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452516+0000", 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452522+0000", 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.088e-06 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452527+0000", 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9420000000000003e-06 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478555+0000", 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026027557 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478630+0000", 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4892000000000001e-05 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478715+0000", 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5436e-05 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7266 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8388608~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897323519, 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026420895999999999, 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.375 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7266 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452533+0000", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452535+0000", 2024-04-23T21:21:42.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7990000000000001e-06 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452541+0000", 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4670000000000003e-06 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452545+0000", 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2089999999999999e-06 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478797+0000", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026252622 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478883+0000", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5390999999999996e-05 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478954+0000", 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1044000000000006e-05 2024-04-23T21:21:42.379 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7267 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9109504~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897304818, 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026543255000000002, 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.380 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7267 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452552+0000", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.41e-07 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452554+0000", 2024-04-23T21:21:42.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.041e-06 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452560+0000", 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1179999999999997e-06 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452565+0000", 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5020000000000003e-06 2024-04-23T21:21:42.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.478989+0000", 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026423965000000001 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479040+0000", 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1697000000000003e-05 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479095+0000", 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4591e-05 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7268 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9306112~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897286402000001, 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026804591999999999, 2024-04-23T21:21:42.385 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7268 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:42.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452570+0000", 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452572+0000", 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8419999999999999e-06 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452578+0000", 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3550000000000004e-06 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452582+0000", 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2780000000000004e-06 2024-04-23T21:21:42.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479186+0000", 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026603907 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479275+0000", 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8918999999999993e-05 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479375+0000", 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9942e-05 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7269 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9502720~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897268531, 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026998115, 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.390 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7269 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452588+0000", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452611+0000", 2024-04-23T21:21:42.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2425e-05 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452617+0000", 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7580000000000003e-06 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452624+0000", 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7109999999999998e-06 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479405+0000", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026780531999999999 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479498+0000", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3045000000000001e-05 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479586+0000", 2024-04-23T21:21:42.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8288999999999994e-05 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7270 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9699328~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897224682000001, 2024-04-23T21:21:42.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027156907000000001, 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7270 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452632+0000", 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:21:42.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452635+0000", 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.187e-06 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452640+0000", 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7910000000000001e-06 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452645+0000", 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8289999999999998e-06 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479616+0000", 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026971222 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479697+0000", 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0463000000000005e-05 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.479789+0000", 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1810000000000004e-05 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7271 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10420224~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.400 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.897204308999999, 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027566067, 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7271 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452652+0000", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452653+0000", 2024-04-23T21:21:42.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452655+0000", 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0559999999999999e-06 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452660+0000", 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6289999999999998e-06 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.452665+0000", 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6430000000000003e-06 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480043+0000", 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027378070000000001 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480130+0000", 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6879999999999995e-05 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.480218+0000", 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8357999999999997e-05 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7388 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6717440~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.405 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891198710999999, 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026122209, 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7388 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458658+0000", 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5200000000000002e-07 2024-04-23T21:21:42.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458660+0000", 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5719999999999999e-06 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458665+0000", 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6609999999999997e-06 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458670+0000", 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3889999999999998e-06 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484666+0000", 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025996717999999999 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484724+0000", 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7222999999999997e-05 2024-04-23T21:21:42.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484780+0000", 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6193999999999998e-05 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7389 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 6914048~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:42.410 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891181179, 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026331738, 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7389 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458675+0000", 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458676+0000", 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7500000000000001e-07 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458678+0000", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6970000000000002e-06 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458684+0000", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5709999999999997e-06 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458688+0000", 2024-04-23T21:21:42.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6099999999999999e-06 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484821+0000", 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026132275 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484908+0000", 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7822e-05 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485007+0000", 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8487999999999996e-05 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7390 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7110656~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891162466000001, 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026475571, 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.415 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7390 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458694+0000", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458702+0000", 2024-04-23T21:21:42.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458708+0000", 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0100000000000001e-06 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458712+0000", 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485062+0000", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026349392999999999 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485112+0000", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0466000000000001e-05 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485170+0000", 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7160999999999997e-05 2024-04-23T21:21:42.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7391 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 7831552~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891137876, 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026684876999999999, 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7391 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:42.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458719+0000", 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7800000000000002e-07 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458722+0000", 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.587e-06 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458728+0000", 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5189999999999998e-06 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458733+0000", 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7140000000000001e-06 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485207+0000", 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026474393999999998 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485300+0000", 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2644000000000003e-05 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485403+0000", 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010364100000000001 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.424 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7392 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8028160~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891114932000001, 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026904931999999999, 2024-04-23T21:21:42.425 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7392 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:42.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458741+0000", 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458742+0000", 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5999999999999999e-07 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458744+0000", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8509999999999999e-06 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458749+0000", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6119999999999998e-06 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458755+0000", 2024-04-23T21:21:42.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1440000000000002e-06 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485459+0000", 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026704828 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485548+0000", 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8985000000000003e-05 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485646+0000", 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8052000000000002e-05 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7393 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 8224768~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.891096177, 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027031339000000001, 2024-04-23T21:21:42.430 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7393 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:42.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.458760+0000", 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460437+0000", 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001677053 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460442+0000", 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460454+0000", 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1704e-05 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460461+0000", 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4909999999999999e-06 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485685+0000", 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025223992000000001 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485736+0000", 2024-04-23T21:21:42.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0633999999999997e-05 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485792+0000", 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5653999999999999e-05 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7397 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9535488~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.889290667000001, 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026115167000000002, 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7397 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460566+0000", 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460568+0000", 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1459999999999999e-06 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460574+0000", 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6720000000000001e-06 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460579+0000", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486493+0000", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.025914311999999998 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486582+0000", 2024-04-23T21:21:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9132e-05 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486681+0000", 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8956000000000006e-05 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7398 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 9732096~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.889271749000001, 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026253736, 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7398 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460585+0000", 2024-04-23T21:21:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460587+0000", 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.035e-06 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460593+0000", 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6670000000000001e-06 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460597+0000", 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4789999999999998e-06 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486723+0000", 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026126018000000001 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486779+0000", 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5445000000000002e-05 2024-04-23T21:21:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486838+0000", 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9704000000000003e-05 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:7399 3.1e 3:79bbec75:::rbd_data.11181344e3fc.0000000000000003:8 [read 10452992~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 12.889253498, 2024-04-23T21:21:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026351192999999998, 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 7399 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460603+0000", 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5699999999999998e-07 2024-04-23T21:21:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460605+0000", 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.781e-06 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460611+0000", 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5509999999999999e-06 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.460615+0000", 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5619999999999997e-06 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486864+0000", 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.026248947000000002 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486911+0000", 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7330000000000003e-05 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486954+0000", 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2664999999999999e-05 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:42.451 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:42.452 INFO:teuthology.orchestra.run.smithi111.stdout:2112 writesame 0xd9053e4 thru 0xd9097d8 data_size 0x3(0x43f5 bytes) 2024-04-23T21:21:42.452 INFO:teuthology.orchestra.run.smithi111.stdout:2113 writesame 0x26b2a9 thru 0x273cfd data_size 0x1(0x8a55 bytes) 2024-04-23T21:21:42.453 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:42.454 INFO:teuthology.orchestra.run.smithi111.stdout:2114 read 0x9a4065f thru 0x9a46234 (0x5bd6 bytes) 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2116 read 0xc518ffe thru 0xc524bd6 (0xbbd9 bytes) 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2117 read 0x608b472 thru 0x6095731 (0xa2c0 bytes) 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2120 trunc from 0xd9097d9 to 0x7dd6b0c 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2121 write 0x1617441 thru 0x1624b2c (0xd6ec bytes) 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2122 punch from 0x3dae4ba to 0x3db8e9b, (0xa9e1 bytes) 2024-04-23T21:21:42.455 INFO:teuthology.orchestra.run.smithi111.stdout:2123 clone 15 order 22 su 131072 sc 12 2024-04-23T21:21:42.456 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:42.452+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:42.553 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:42.553+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:42.557 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:42.557 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:42.566 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:42.654 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:42.654+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:42.709 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:42.709 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:42.709 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:42.709 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:42.709 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:42.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:42.758 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:42.759+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:42.856 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:42.856+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3137 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3145728~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444714118, 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014041269, 2024-04-23T21:21:42.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3137 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417388+0000", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417394+0000", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.55e-06 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417401+0000", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5930000000000002e-06 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417406+0000", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5059999999999997e-06 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431251+0000", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013845245000000001 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431327+0000", 2024-04-23T21:21:42.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6409000000000007e-05 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431429+0000", 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101582 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3138 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3211264~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.44468964, 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014240901, 2024-04-23T21:21:42.866 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3138 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417412+0000", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8099999999999998e-07 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417419+0000", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8979999999999999e-06 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417426+0000", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5100000000000004e-06 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417431+0000", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7940000000000002e-06 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431471+0000", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014040336 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431551+0000", 2024-04-23T21:21:42.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0343e-05 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431653+0000", 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010163899999999999 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3139 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3276800~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444664687, 2024-04-23T21:21:42.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014372180999999999, 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3139 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417437+0000", 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417439+0000", 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8950000000000001e-06 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417453+0000", 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3383e-05 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417458+0000", 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431688+0000", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014230456000000001 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431755+0000", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7183999999999999e-05 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431809+0000", 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3816000000000001e-05 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.874 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3140 3.9 3:9122d7cd:::rbd_data.1118c9f76dd8.0000000000000003:4 [read 3342336~32768] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.44463687, 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014584774999999999, 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.875 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3140 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417465+0000", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417467+0000", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417473+0000", 2024-04-23T21:21:42.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2559999999999999e-06 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417478+0000", 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3510000000000004e-06 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431863+0000", 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014384868 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.431944+0000", 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1433000000000005e-05 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432050+0000", 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105503 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3141 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444614799, 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014710842999999999, 2024-04-23T21:21:42.879 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3141 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417487+0000", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417489+0000", 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7269999999999999e-06 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417495+0000", 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2060000000000004e-06 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417506+0000", 2024-04-23T21:21:42.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1277000000000001e-05 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432144+0000", 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014637281 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432198+0000", 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3965999999999998e-05 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3142 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444588322000001, 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014717164, 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3142 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417513+0000", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417514+0000", 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417516+0000", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.875e-06 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417522+0000", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4169999999999997e-06 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417526+0000", 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4410000000000003e-06 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432223+0000", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014696300000000001 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432231+0000", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7409999999999992e-06 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3143 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:42.887 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444568819000001, 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730144000000001, 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3143 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417533+0000", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417535+0000", 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7090000000000001e-06 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417541+0000", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2110000000000003e-06 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417551+0000", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9250000000000007e-06 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432251+0000", 2024-04-23T21:21:42.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014700210999999999 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432263+0000", 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1715999999999999e-05 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3144 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444543969000001, 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014732579000000001, 2024-04-23T21:21:42.891 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3144 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417558+0000", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.46e-07 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417560+0000", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.945e-06 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417566+0000", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1909999999999997e-06 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417571+0000", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432283+0000", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014711864999999999 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432290+0000", 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.525e-06 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3145 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444524604000001, 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014738942, 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3145 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417577+0000", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417580+0000", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.125e-06 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417586+0000", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3339999999999998e-06 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417591+0000", 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6809999999999997e-06 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432309+0000", 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014718916 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432316+0000", 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3146 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444497290000001, 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014741944, 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3146 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417604+0000", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417605+0000", 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417607+0000", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417612+0000", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5079999999999996e-06 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417617+0000", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1919999999999998e-06 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432340+0000", 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014723234 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432346+0000", 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4910000000000002e-06 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3147 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444478438000001, 2024-04-23T21:21:42.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014787737, 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3147 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417623+0000", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417624+0000", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417626+0000", 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.012e-06 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417632+0000", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0220000000000001e-06 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417636+0000", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3000000000000003e-06 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432366+0000", 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014730115 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432411+0000", 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4895000000000003e-05 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3148 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444459784999999, 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014811695999999999, 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3148 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417642+0000", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6100000000000002e-07 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417650+0000", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1019999999999997e-06 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417658+0000", 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1189999999999997e-06 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417664+0000", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432443+0000", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014779336000000001 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432454+0000", 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0713999999999999e-05 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3149 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444430918999998, 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014815238, 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3149 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417671+0000", 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3500000000000002e-07 2024-04-23T21:21:42.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417673+0000", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417679+0000", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4530000000000001e-06 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417683+0000", 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2560000000000004e-06 2024-04-23T21:21:42.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432479+0000", 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014796236000000001 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432486+0000", 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7479999999999996e-06 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3150 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.913 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.44441247, 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014825331000000001, 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3150 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:42.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417689+0000", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417690+0000", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1600000000000001e-07 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417696+0000", 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3169999999999998e-06 2024-04-23T21:21:42.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417703+0000", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1729999999999998e-06 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417708+0000", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.481e-06 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432508+0000", 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014799827999999999 2024-04-23T21:21:42.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432515+0000", 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0160000000000003e-06 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3151 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444387486, 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014864755, 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.917 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3151 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417714+0000", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417715+0000", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417717+0000", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2500000000000001e-06 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417724+0000", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.289e-06 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417729+0000", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650000000000002e-06 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432567+0000", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014837845000000001 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432579+0000", 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2334000000000001e-05 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3152 3.9 3.b3eb4489 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.444366511999998, 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014901721, 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3152 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417735+0000", 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417736+0000", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.96e-07 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417738+0000", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0540000000000002e-06 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417743+0000", 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2510000000000002e-06 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.417755+0000", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2536e-05 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432629+0000", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014873569 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.432637+0000", 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9149999999999996e-06 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.924 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:4908 3.3 3:c0665d5c:::rbd_data.11181344e3fc.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 11010048] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.675097748000001, 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030900347000000002, 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 4908 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187004+0000", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.187005+0000", 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9800000000000002e-07 2024-04-23T21:21:42.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198009+0000", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011004491999999999 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198032+0000", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2753000000000001e-05 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198059+0000", 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6792e-05 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198122+0000", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3633999999999994e-05 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.198236+0000", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011307 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.199108+0000", 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00087266500000000001 2024-04-23T21:21:42.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.210413+0000", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011304611000000001 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217815+0000", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0074019869999999996 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217856+0000", 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1411e-05 2024-04-23T21:21:42.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.217904+0000", 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8034000000000001e-05 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4935 3.1b e26/18 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:head v 26'8, mlcod=24'5)", 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.619145682999999, 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024520146999999999, 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.930 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242956+0000", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.242957+0000", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7800000000000001e-07 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259689+0000", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016732883 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259714+0000", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4992000000000001e-05 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259726+0000", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1916000000000001e-05 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259814+0000", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7964e-05 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.259834+0000", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0083999999999999e-05 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267434+0000", 2024-04-23T21:21:42.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0075992239999999999 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.267476+0000", 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2506000000000003e-05 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:4953 3.1b e26/18 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:head v 26'9, mlcod=25'7)", 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.58232604, 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024344686000000001, 2024-04-23T21:21:42.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279776+0000", 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.279777+0000", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.75e-06 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295820+0000", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016042178000000001 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295845+0000", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5225e-05 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295854+0000", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.064e-06 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295951+0000", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6806999999999997e-05 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.295969+0000", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8672999999999999e-05 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:42.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304074+0000", 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081043190000000005 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:28.304120+0000", 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6669999999999999e-05 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:13160 3.14 3:2914a515:::rbd_data.11188429226.0000000000000017:head [set-alloc-hint object_size 8388608 write_size 8388608,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:37.362674+0000", 2024-04-23T21:21:42.938 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4994272180000001, 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029037004000000002, 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 13160 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362674+0000", 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362674+0000", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362676+0000", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.658e-06 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362683+0000", 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1960000000000002e-06 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362699+0000", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5741999999999999e-05 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362708+0000", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4469999999999995e-06 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362759+0000", 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0793e-05 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.362851+0000", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1990000000000005e-05 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 7", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.367506+0000", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046547109999999997 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.391574+0000", 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024067848999999999 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.391630+0000", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6586000000000003e-05 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.391672+0000", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1355e-05 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:37.391711+0000", 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9677e-05 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.943 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:42.944 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:42.944 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:42.944 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:21:42.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:42.957 INFO:teuthology.orchestra.run.smithi111.stdout:flattening image image_client.0-clone14 2024-04-23T21:21:42.958 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:42.958+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:43.055 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-04-23T21:21:43.059 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:43.058+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:43.089 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.089 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:43.089 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:43.089 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:43.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:43.181 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:43.181+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:43.239 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.240 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:43.240 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:43.240 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:43.240 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:43.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:43.254 INFO:teuthology.orchestra.run.smithi111.stdout:2124 writesame 0x38b9ad2 thru 0x38c45bf data_size 0x3(0xaaee bytes) 2024-04-23T21:21:43.260 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:43.260+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:43.267 INFO:teuthology.orchestra.run.smithi111.stdout:2125 read 0x3511704 thru 0x3511bf4 (0x4f1 bytes) 2024-04-23T21:21:43.267 INFO:teuthology.orchestra.run.smithi111.stdout:2127 punch from 0x66567b3 to 0x665bad2, (0x531f bytes) 2024-04-23T21:21:43.267 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:43.267 INFO:teuthology.orchestra.run.smithi111.stdout:2129 read 0x35ecd38 thru 0x35f1dc9 (0x5092 bytes) 2024-04-23T21:21:43.274 INFO:teuthology.orchestra.run.smithi111.stdout:2130 trunc from 0x7dd6b0c to 0x22cc47a 2024-04-23T21:21:43.318 INFO:teuthology.orchestra.run.smithi111.stdout:2131 clone 16 order 19 su 2048 sc 11 2024-04-23T21:21:43.361 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:43.361+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:43.399 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-04-23T21:21:43.399 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-04-23T21:21:43.399 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-23T21:21:43.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2140 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7372800~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:43.401 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118337832000002, 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032102877000000002, 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2140 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:43.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282177+0000", 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0379999999999998e-06 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282185+0000", 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9119999999999998e-06 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282190+0000", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7269999999999998e-06 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314126+0000", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031936101000000001 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314204+0000", 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7791000000000001e-05 2024-04-23T21:21:43.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314278+0000", 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3696999999999996e-05 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:43.406 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118311838, 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:43.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:21:43.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:43.411 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118291760000002, 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:43.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:21:43.416 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118271131, 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.417 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:21:43.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.422 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118245872999999, 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118219293999999, 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:21:43.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.432 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118199140000002, 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.433 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118173380000002, 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.438 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:21:43.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.442 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118142090999999, 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.443 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:21:43.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.447 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118127523999998, 2024-04-23T21:21:43.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:21:43.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:21:43.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.456 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:43.457 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118108108000001, 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:21:43.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118092924999999, 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:21:43.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:43.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:21:43.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:21:43.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118082355999999, 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.469 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118072738999999, 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.475 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:21:43.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:21:43.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:21:43.479 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118063206999999, 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.480 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:43.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:21:43.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:21:43.484 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.118050209, 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:21:43.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:21:43.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.991004251, 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:21:43.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.990985505000001, 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:21:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:21:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.990949514, 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:21:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:21:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.990921931999999, 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:21:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:21:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:21:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.511 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:43.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:43.514 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:43.513+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:43.514 INFO:tasks.ceph.osd.1:Started 2024-04-23T21:21:43.515 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:21:43.612 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:43.612 INFO:teuthology.orchestra.run.smithi111.stderr:no valid command found; 10 closest matches: 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:0 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:1 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:2 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:abort 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:assert 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:config diff 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:config diff get 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:config get 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:config help [] 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:config set ... 2024-04-23T21:21:43.613 INFO:teuthology.orchestra.run.smithi111.stderr:admin_socket: invalid command 2024-04-23T21:21:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T21:21:43.616 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-04-23T21:21:43.616 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:43.615+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:43.656 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.656 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:43.657 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:43.657 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:43.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:43.715 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:43.715+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:43.730 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:43.731 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:43.731 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:43.748 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:43.748 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:43.765 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:43.765 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:43.767 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.767 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:43.767 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:43.767 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:43.767 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:43.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:43.782 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:43.782 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:43.816 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:43.817+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:43.907 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:21:43.917 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:43.917+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:43.924 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:2565 3.1e e23/18 3:7adf3bd0:::rbd_data.11183c92f5c.0000000000000005:head v 23'23, mlcod=22'21)", 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 19.553226605999999, 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063423730000000001, 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.74e-07 2024-04-23T21:21:43.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374893+0000", 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038888899999999999 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374909+0000", 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:21:43.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374921+0000", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2031000000000001e-05 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374987+0000", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5827000000000005e-05 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.375003+0000", 2024-04-23T21:21:43.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6393e-05 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377299+0000", 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002295503 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377346+0000", 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7475000000000002e-05 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:8046 3.a 3:56ba5b24:::rbd_data.111830ac55a1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.681284801, 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014462938, 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 8046 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242952+0000", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4650000000000002e-06 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242966+0000", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3708e-05 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242974+0000", 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0369999999999995e-06 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243016+0000", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2769000000000003e-05 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243063+0000", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999997e-05 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.245737+0000", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026734939999999998 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257269+0000", 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011532351 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257289+0000", 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0605e-05 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257352+0000", 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2535999999999999e-05 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257408+0000", 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6450000000000003e-05 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10825 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6857738720000004, 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0062412700000000002, 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10825 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238457+0000", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3099999999999996e-07 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238463+0000", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5090000000000001e-06 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238474+0000", 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1455e-05 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238484+0000", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5480000000000007e-06 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240357+0000", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001872964 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240446+0000", 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9569999999999998e-05 2024-04-23T21:21:43.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240766+0000", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031934500000000001 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244476+0000", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037105229999999999 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244570+0000", 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3869999999999994e-05 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244641+0000", 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0737999999999999e-05 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244698+0000", 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6817000000000002e-05 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10826 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6856492190000001, 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080318590000000006, 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10826 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:43.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238582+0000", 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0399999999999995e-07 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238586+0000", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2740000000000001e-06 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238595+0000", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1220000000000001e-06 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238602+0000", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3559999999999998e-06 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241310+0000", 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027081649999999998 2024-04-23T21:21:43.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241371+0000", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0829000000000003e-05 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241647+0000", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027606199999999999 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244483+0000", 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028360680000000002 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246556+0000", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002073713 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246588+0000", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1829000000000003e-05 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246613+0000", 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4737e-05 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.945 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10827 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6856112700000008, 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008083504, 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.946 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10827 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238620+0000", 2024-04-23T21:21:43.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-07 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238622+0000", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6479999999999999e-06 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238628+0000", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3489999999999999e-06 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238631+0000", 2024-04-23T21:21:43.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2990000000000001e-06 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242043+0000", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034116889999999999 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242092+0000", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9542000000000001e-05 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242319+0000", 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022715600000000001 2024-04-23T21:21:43.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245447+0000", 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031279509999999999 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246652+0000", 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001204526 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246681+0000", 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8892999999999999e-05 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246703+0000", 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1869999999999999e-05 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10828 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.951 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6855856960000004, 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011309565000000001, 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10828 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-07 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238648+0000", 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8619999999999999e-06 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238656+0000", 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0909999999999993e-06 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238661+0000", 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2349999999999999e-06 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242733+0000", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040721500000000001 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242819+0000", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5457999999999994e-05 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243185+0000", 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036640300000000001 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246517+0000", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003332149 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249839+0000", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033218850000000001 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249896+0000", 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7085e-05 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249954+0000", 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7782999999999999e-05 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10829 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6855591289999996, 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011463991999999999, 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.957 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10829 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:43.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238672+0000", 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8599999999999998e-07 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238674+0000", 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6639999999999998e-06 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238681+0000", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1639999999999998e-06 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238689+0000", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0800000000000006e-06 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243618+0000", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049292119999999997 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243705+0000", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7101000000000005e-05 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244084+0000", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037872300000000003 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246524+0000", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002440331 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250022+0000", 2024-04-23T21:21:43.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034982429999999998 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250081+0000", 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8403000000000003e-05 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250135+0000", 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4585e-05 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10830 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6854541750000003, 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011515780999999999, 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10830 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238777+0000", 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0200000000000001e-07 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238781+0000", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238790+0000", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1169999999999993e-06 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238797+0000", 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8419999999999999e-06 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244765+0000", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059680519999999997 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244848+0000", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2145999999999996e-05 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245196+0000", 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034871799999999998 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246712+0000", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015162190000000001 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250197+0000", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034843389999999999 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250250+0000", 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3035999999999998e-05 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250292+0000", 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2120999999999999e-05 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10831 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6854089170000002, 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012631310999999999, 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10831 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238822+0000", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7800000000000002e-07 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238827+0000", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4310000000000002e-06 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238837+0000", 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4010000000000003e-06 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238843+0000", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2079999999999997e-06 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245485+0000", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0066425499999999997 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245536+0000", 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0149999999999999e-05 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245733+0000", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000197231 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246718+0000", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00098556699999999991 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251355+0000", 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046364290000000001 2024-04-23T21:21:43.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251406+0000", 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1038999999999997e-05 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251453+0000", 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6826999999999997e-05 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10832 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.973 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6853598880000007, 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012899838, 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10832 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238871+0000", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3200000000000005e-07 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238874+0000", 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4429999999999999e-06 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238883+0000", 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6859999999999995e-06 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238891+0000", 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9889999999999992e-06 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246016+0000", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071248190000000001 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246065+0000", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8872999999999998e-05 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246281+0000", 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000216393 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249969+0000", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036876690000000002 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251663+0000", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016940519999999999 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251725+0000", 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1818999999999999e-05 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251770+0000", 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5562999999999997e-05 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10833 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6846748700000003, 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012352699999999999, 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.979 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10833 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:43.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239556+0000", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239562+0000", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1610000000000002e-06 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239572+0000", 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0738e-05 2024-04-23T21:21:43.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239579+0000", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1099999999999997e-06 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246746+0000", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071668790000000001 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246798+0000", 2024-04-23T21:21:43.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1748e-05 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246995+0000", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019731800000000001 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249976+0000", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002981058 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251830+0000", 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00185395 2024-04-23T21:21:43.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251884+0000", 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3171999999999998e-05 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251908+0000", 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4652999999999999e-05 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10834 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6842108059999994, 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012040969, 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10834 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:43.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8599999999999997e-07 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240034+0000", 2024-04-23T21:21:43.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3552e-05 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240046+0000", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1924e-05 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240052+0000", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-06 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247246+0000", 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071939179999999997 2024-04-23T21:21:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247321+0000", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5110000000000004e-05 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247520+0000", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000198743 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250307+0000", 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002786462 2024-04-23T21:21:43.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251964+0000", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001657309 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252019+0000", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4728000000000002e-05 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252060+0000", 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1847000000000001e-05 2024-04-23T21:21:43.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10835 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6839689839999998, 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011966503999999999, 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10835 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:43.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240262+0000", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240268+0000", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7370000000000003e-06 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240279+0000", 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1569000000000001e-05 2024-04-23T21:21:43.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240287+0000", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.926e-06 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247891+0000", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0076038410000000001 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247988+0000", 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6704000000000001e-05 2024-04-23T21:21:43.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248469+0000", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00048112900000000001 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250313+0000", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018440119999999999 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252122+0000", 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018091990000000001 2024-04-23T21:21:43.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252178+0000", 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5603000000000003e-05 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252228+0000", 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9985000000000003e-05 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.995 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10836 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6834409949999998, 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011591181000000001, 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10836 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:43.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240790+0000", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7400000000000002e-07 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:43.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240795+0000", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4689999999999999e-06 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240803+0000", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4940000000000003e-06 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240808+0000", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9729999999999997e-06 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:43.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248916+0000", 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081076729999999993 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249008+0000", 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2650000000000002e-05 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249420+0000", 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041150699999999998 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:43.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250324+0000", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00090375899999999996 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252289+0000", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019649149999999998 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252343+0000", 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3977999999999997e-05 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252381+0000", 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10839 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.6824437359999997, 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010757071999999999, 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.001 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10839 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9999999999999997e-07 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241792+0000", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4460000000000003e-06 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241804+0000", 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.257e-05 2024-04-23T21:21:44.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241812+0000", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3089999999999998e-06 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250357+0000", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0085456630000000002 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250606+0000", 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024869900000000001 2024-04-23T21:21:44.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250982+0000", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037560399999999998 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252243+0000", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001261306 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252436+0000", 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019354800000000001 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252502+0000", 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5513999999999996e-05 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252544+0000", 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1712999999999999e-05 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:18898 3.a 3:5512b404:::rbd_data.11181e0f6ea.0000000000000058:head [truncate 163840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:44.006 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.2816271600000002, 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063585339999999999, 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 18898 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:44.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642604+0000", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642607+0000", 2024-04-23T21:21:44.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.836e-06 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642613+0000", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642618+0000", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3730000000000001e-06 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643185+0000", 2024-04-23T21:21:44.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056650699999999995 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643263+0000", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8778999999999999e-05 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643395+0000", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013110699999999999 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.645694+0000", 2024-04-23T21:21:44.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022991880000000002 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648900+0000", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032063669999999999 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648937+0000", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6613999999999999e-05 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648962+0000", 2024-04-23T21:21:44.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4953999999999999e-05 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19031 3.a 3:50c752db:::rbd_data.11181e0f6ea.0000000000000057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.1525743749999999, 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070230910000000004, 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19031 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5400000000000002e-07 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771658+0000", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7099999999999999e-06 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771662+0000", 2024-04-23T21:21:44.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4630000000000003e-06 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771672+0000", 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.115e-06 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772211+0000", 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053961700000000005 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772288+0000", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6878000000000005e-05 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772401+0000", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011273300000000001 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.773095+0000", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00069380099999999997 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778585+0000", 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054904460000000004 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778641+0000", 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5720000000000002e-05 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778679+0000", 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8254000000000003e-05 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19032 3.a 3:5512b404:::rbd_data.11181e0f6ea.0000000000000058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.1525490060000001, 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072067279999999999, 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19032 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771682+0000", 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771684+0000", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771691+0000", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8329999999999996e-06 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771696+0000", 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2059999999999998e-06 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772660+0000", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096427100000000005 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772724+0000", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3350999999999997e-05 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772825+0000", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101366 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778690+0000", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058654750000000002 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778781+0000", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0169999999999999e-05 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778839+0000", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8674000000000001e-05 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778888+0000", 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8779999999999997e-05 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19050 3.a 3:5676f4c4:::rbd_data.11181e0f6ea.000000000000008f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.1454163400000001, 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055626549999999997, 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19050 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:44.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778815+0000", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1829999999999999e-06 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778820+0000", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8910000000000003e-06 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778832+0000", 2024-04-23T21:21:44.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1499e-05 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778839+0000", 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5209999999999997e-06 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778930+0000", 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1051999999999995e-05 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779020+0000", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9536000000000005e-05 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779126+0000", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105986 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.781426+0000", 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023008270000000001 2024-04-23T21:21:44.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784299+0000", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028725930000000001 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784351+0000", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2309999999999999e-05 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784377+0000", 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5256999999999999e-05 2024-04-23T21:21:44.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19054 3.a 3:57d3f192:::rbd_data.11181e0f6ea.00000000000000b3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 2.145276339, 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055262030000000004, 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:44.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19054 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778955+0000", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5300000000000003e-07 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778959+0000", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4490000000000001e-06 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778970+0000", 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1161999999999999e-05 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778979+0000", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2349999999999999e-06 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779424+0000", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044492400000000002 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779505+0000", 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1559999999999998e-05 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779637+0000", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132395 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784172+0000", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045345419999999999 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784411+0000", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000238994 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:44.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784441+0000", 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0261999999999999e-05 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784480+0000", 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8927000000000002e-05 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:44.034 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:44.035 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone15 from 0x22cc47a (overlap 0x22cc47a) to 0x37ad3f 2024-04-23T21:21:44.036 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:44.036 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:44.036 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:44.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:44.037 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:44.035+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:44.037 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:44.038 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-04-23T21:21:44.039 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:44.040 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:44.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:44.102 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:44.102+0000 7f4a10b2d640 -1 Falling back to public interface 2024-04-23T21:21:44.136 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:44.136+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:44.166 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:44.167 INFO:teuthology.orchestra.run.smithi111.stdout:2135 punch from 0x1629642 to 0x16374c9, (0xde87 bytes) 2024-04-23T21:21:44.171 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:44.179 INFO:teuthology.orchestra.run.smithi111.stdout:2137 write 0x3a78e80 thru 0x3a84952 (0xbad3 bytes) 2024-04-23T21:21:44.182 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.182 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:44.195 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:44.195 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:44.195 INFO:teuthology.orchestra.run.smithi111.stdout:2140 read 0x194819f thru 0x1949380 (0x11e2 bytes) 2024-04-23T21:21:44.195 INFO:teuthology.orchestra.run.smithi111.stdout:2143 clone 17 order 18 su 8192 sc 15 2024-04-23T21:21:44.237 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:44.236+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:44.336 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.336 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.337 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:44.336+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:44.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:44.437 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:44.438+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:44.488 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.488 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:44.538 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:44.539+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.600 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.608 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:44.652 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:44.652+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:44.710 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.711 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:44.760 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:44.761+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:44.841 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:44.842+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:44.861 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.861 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:44.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:44.942 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:44.943+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:44.979 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone16 intact 2024-04-23T21:21:44.988 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:44.988 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:45.043 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:45.044+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:45.066 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-04-23T21:21:45.093 INFO:teuthology.orchestra.run.smithi111.stdout:2144 write 0xa9d187a thru 0xa9d641e (0x4ba5 bytes) 2024-04-23T21:21:45.102 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:45.113 INFO:teuthology.orchestra.run.smithi111.stdout:2147 trunc from 0xa9d641f to 0x1675713 2024-04-23T21:21:45.129 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.129 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:45.144 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:45.145+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:45.194 INFO:teuthology.orchestra.run.smithi111.stdout:2148 write 0xd6d3e12 thru 0xd6d626f (0x245e bytes) 2024-04-23T21:21:45.201 INFO:teuthology.orchestra.run.smithi111.stdout:2149 writesame 0x9d87c90 thru 0x9d8d49b data_size 0x73(0x580c bytes) 2024-04-23T21:21:45.210 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:45.217 INFO:teuthology.orchestra.run.smithi111.stdout:2151 trunc from 0xd6d6270 to 0x776e041 2024-04-23T21:21:45.231 INFO:teuthology.orchestra.run.smithi111.stdout:2152 writesame 0x3adb0f1 thru 0x3adf579 data_size 0x37(0x4489 bytes) 2024-04-23T21:21:45.239 INFO:teuthology.orchestra.run.smithi111.stdout:2153 punch from 0x583cfd9 to 0x584726e, (0xa295 bytes) 2024-04-23T21:21:45.239 INFO:teuthology.orchestra.run.smithi111.stdout:2154 writesame 0x54af962 thru 0x54bd387 data_size 0x7(0xda26 bytes) 2024-04-23T21:21:45.245 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:45.245+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:45.251 INFO:teuthology.orchestra.run.smithi111.stdout:2155 read 0xd879d5 thru 0xd8f913 (0x7f3f bytes) 2024-04-23T21:21:45.254 INFO:teuthology.orchestra.run.smithi111.stdout:2156 write 0x6e8a99e thru 0x6e90b0e (0x6171 bytes) 2024-04-23T21:21:45.261 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.262 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.263 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:45.271 INFO:teuthology.orchestra.run.smithi111.stdout:2158 writesame 0x97e5181 thru 0x97e708f data_size 0x1(0x1f0f bytes) 2024-04-23T21:21:45.271 INFO:teuthology.orchestra.run.smithi111.stdout:2159 read 0x29842a8 thru 0x298737a (0x30d3 bytes) 2024-04-23T21:21:45.271 INFO:teuthology.orchestra.run.smithi111.stdout:2160 read 0x871c77f thru 0x8723768 (0x6fea bytes) 2024-04-23T21:21:45.274 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:45.282 INFO:teuthology.orchestra.run.smithi111.stdout:2161 trunc from 0x97e7090 to 0x434fd2 2024-04-23T21:21:45.346 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:45.346+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:45.380 INFO:teuthology.orchestra.run.smithi111.stdout:2162 trunc from 0x434fd2 to 0x728e76d 2024-04-23T21:21:45.399 INFO:teuthology.orchestra.run.smithi111.stdout:2163 write 0xd85e936 thru 0xd868136 (0x9801 bytes) 2024-04-23T21:21:45.411 INFO:teuthology.orchestra.run.smithi111.stdout:2166 read 0x74e794a thru 0x74eb7ec (0x3ea3 bytes) 2024-04-23T21:21:45.421 INFO:teuthology.orchestra.run.smithi111.stdout:2167 trunc from 0xd868137 to 0x4727eac 2024-04-23T21:21:45.422 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.422 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:45.433 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:45.433 INFO:teuthology.orchestra.run.smithi111.stdout:2171 clone 18 order 22 su 262144 sc 7 2024-04-23T21:21:45.446 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:45.447+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:45.533 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.533 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.542 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:45.548 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:45.548+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:45.644 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.645 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.649 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:45.650+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:45.653 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:45.750 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:45.751+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:45.756 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.756 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.765 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:45.850 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:45.851+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:45.866 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:45.866 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:45.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:45.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:21:45.959 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:45.960+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:45.986 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone17 from 0x4727eac (overlap 0x434fd2) to 0x0 2024-04-23T21:21:46.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:46.048 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:46.048 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:46.059 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:46.088 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:46.089+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:46.153 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:46.154+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:46.161 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-04-23T21:21:46.188 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:46.188 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:46.189 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:46.230 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:46.230 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:46.234 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:46.234 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:46.249 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:46.266 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:46.267 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:46.291 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:46.292+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:46.308 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:46.308 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:46.338 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:46.338 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:46.355 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:46.356+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:46.367 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:46.367 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:46.395 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:46.395 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:46.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:46.448 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:46.448 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:46.448 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:46.448 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:46.448 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:46.456 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:46.457+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:46.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:46.557 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:46.558+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:46.657 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:46.659+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:46.658 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2140 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7372800~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374177940999999, 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032102877000000002, 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2140 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282177+0000", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0379999999999998e-06 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282185+0000", 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9119999999999998e-06 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282190+0000", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7269999999999998e-06 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314126+0000", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031936101000000001 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314204+0000", 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7791000000000001e-05 2024-04-23T21:21:46.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314278+0000", 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3696999999999996e-05 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374151947000001, 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.667 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374131868999999, 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:46.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:21:46.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374111240000001, 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374085982, 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:46.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:21:46.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374059403, 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.680 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:21:46.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374039248999999, 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:21:46.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:46.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:21:46.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:21:46.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.374013488999999, 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.3739822, 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.692 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:21:46.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373967632999999, 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.696 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:46.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:21:46.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:21:46.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373948216999999, 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.700 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:46.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:21:46.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373933034, 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:21:46.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:21:46.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373922465, 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373912848, 2024-04-23T21:21:46.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:46.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373903316, 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:21:46.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:21:46.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:21:46.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.373890318000001, 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.246844360000001, 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.727 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:46.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:21:46.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:21:46.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.246825613999999, 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.731 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:21:46.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.246789623000002, 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:21:46.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:21:46.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:21:46.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:21:46.739 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.246762041, 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:21:46.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:46.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:21:46.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:21:46.743 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:46.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:46.744 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:46.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:46.788 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:46.789+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:46.860 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:46.860+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:46.902 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:46.902 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:46.902 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:46.902 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:46.915 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:46.923 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:21:47.000 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:47.001+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:47.061 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:47.062+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:47.104 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:47.104 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:47.104 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:47.104 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:47.104 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:47.113 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:47.162 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:47.163+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:47.262 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:2565 3.1e e23/18 3:7adf3bd0:::rbd_data.11183c92f5c.0000000000000005:head v 23'23, mlcod=22'21)", 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 22.890876969000001, 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063423730000000001, 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.371004+0000", 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.74e-07 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374893+0000", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0038888899999999999 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374909+0000", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374921+0000", 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2031000000000001e-05 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.374987+0000", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5827000000000005e-05 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.375003+0000", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6393e-05 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377299+0000", 2024-04-23T21:21:47.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002295503 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.377346+0000", 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7475000000000002e-05 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:8046 3.a 3:56ba5b24:::rbd_data.111830ac55a1.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:47.267 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 17.018935163999998, 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014462938, 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 8046 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:47.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242946+0000", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0299999999999997e-07 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242952+0000", 2024-04-23T21:21:47.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4650000000000002e-06 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242966+0000", 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3708e-05 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.242974+0000", 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0369999999999995e-06 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243016+0000", 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2769000000000003e-05 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.243063+0000", 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999997e-05 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.245737+0000", 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026734939999999998 2024-04-23T21:21:47.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257269+0000", 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011532351 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257289+0000", 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0605e-05 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257352+0000", 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2535999999999999e-05 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:30.257408+0000", 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6450000000000003e-05 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10825 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.273 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023424235, 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0062412700000000002, 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10825 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238456+0000", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238457+0000", 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3099999999999996e-07 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238463+0000", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5090000000000001e-06 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238474+0000", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1455e-05 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238484+0000", 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5480000000000007e-06 2024-04-23T21:21:47.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240357+0000", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001872964 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240446+0000", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9569999999999998e-05 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240766+0000", 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00031934500000000001 2024-04-23T21:21:47.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244476+0000", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0037105229999999999 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244570+0000", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3869999999999994e-05 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244641+0000", 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0737999999999999e-05 2024-04-23T21:21:47.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244698+0000", 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6817000000000002e-05 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10826 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023299582, 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0080318590000000006, 2024-04-23T21:21:47.279 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10826 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238581+0000", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238582+0000", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0399999999999995e-07 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238586+0000", 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2740000000000001e-06 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238595+0000", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1220000000000001e-06 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238602+0000", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3559999999999998e-06 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241310+0000", 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0027081649999999998 2024-04-23T21:21:47.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241371+0000", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0829000000000003e-05 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241647+0000", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00027606199999999999 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244483+0000", 2024-04-23T21:21:47.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028360680000000002 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246556+0000", 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002073713 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246588+0000", 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1829000000000003e-05 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246613+0000", 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4737e-05 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10827 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023261633000001, 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008083504, 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10827 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238619+0000", 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238620+0000", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-07 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238622+0000", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6479999999999999e-06 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238628+0000", 2024-04-23T21:21:47.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3489999999999999e-06 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238631+0000", 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2990000000000001e-06 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242043+0000", 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034116889999999999 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242092+0000", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9542000000000001e-05 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242319+0000", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00022715600000000001 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245447+0000", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0031279509999999999 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246652+0000", 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001204526 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246681+0000", 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8892999999999999e-05 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246703+0000", 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1869999999999999e-05 2024-04-23T21:21:47.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10828 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023236059, 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011309565000000001, 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10828 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238645+0000", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-07 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238648+0000", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8619999999999999e-06 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238656+0000", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0909999999999993e-06 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238661+0000", 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2349999999999999e-06 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242733+0000", 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0040721500000000001 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.242819+0000", 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5457999999999994e-05 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243185+0000", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00036640300000000001 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246517+0000", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003332149 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249839+0000", 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033218850000000001 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249896+0000", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7085e-05 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249954+0000", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7782999999999999e-05 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10829 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023209491999999, 2024-04-23T21:21:47.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011463991999999999, 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10829 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238671+0000", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238672+0000", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8599999999999998e-07 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238674+0000", 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6639999999999998e-06 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238681+0000", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1639999999999998e-06 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238689+0000", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0800000000000006e-06 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243618+0000", 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049292119999999997 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.243705+0000", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7101000000000005e-05 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244084+0000", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037872300000000003 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246524+0000", 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002440331 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250022+0000", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034982429999999998 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250081+0000", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8403000000000003e-05 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250135+0000", 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4585e-05 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10830 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023104538, 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011515780999999999, 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10830 2024-04-23T21:21:47.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238776+0000", 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238777+0000", 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0200000000000001e-07 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238781+0000", 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238790+0000", 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1169999999999993e-06 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238797+0000", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8419999999999999e-06 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244765+0000", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059680519999999997 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.244848+0000", 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2145999999999996e-05 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245196+0000", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00034871799999999998 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246712+0000", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015162190000000001 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250197+0000", 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0034843389999999999 2024-04-23T21:21:47.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250250+0000", 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3035999999999998e-05 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250292+0000", 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2120999999999999e-05 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.307 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10831 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.02305928, 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012631310999999999, 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10831 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238821+0000", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238822+0000", 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7800000000000002e-07 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238827+0000", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4310000000000002e-06 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238837+0000", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4010000000000003e-06 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238843+0000", 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2079999999999997e-06 2024-04-23T21:21:47.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245485+0000", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0066425499999999997 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245536+0000", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0149999999999999e-05 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.245733+0000", 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000197231 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246718+0000", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00098556699999999991 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251355+0000", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0046364290000000001 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251406+0000", 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1038999999999997e-05 2024-04-23T21:21:47.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251453+0000", 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6826999999999997e-05 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10832 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.023010251000001, 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012899838, 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.313 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10832 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238870+0000", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238871+0000", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3200000000000005e-07 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238874+0000", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4429999999999999e-06 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238883+0000", 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6859999999999995e-06 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.238891+0000", 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9889999999999992e-06 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246016+0000", 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071248190000000001 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246065+0000", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8872999999999998e-05 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246281+0000", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000216393 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249969+0000", 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0036876690000000002 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251663+0000", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016940519999999999 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251725+0000", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1818999999999999e-05 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251770+0000", 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5562999999999997e-05 2024-04-23T21:21:47.318 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10833 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.022325233, 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012352699999999999, 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10833 2024-04-23T21:21:47.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239555+0000", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239556+0000", 2024-04-23T21:21:47.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239562+0000", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1610000000000002e-06 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239572+0000", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0738e-05 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.239579+0000", 2024-04-23T21:21:47.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1099999999999997e-06 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246746+0000", 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071668790000000001 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246798+0000", 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1748e-05 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.246995+0000", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019731800000000001 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249976+0000", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002981058 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251830+0000", 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00185395 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251884+0000", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3171999999999998e-05 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251908+0000", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4652999999999999e-05 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10834 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:47.324 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.021861168999999, 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012040969, 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10834 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240020+0000", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8599999999999997e-07 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240034+0000", 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3552e-05 2024-04-23T21:21:47.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240046+0000", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1924e-05 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240052+0000", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-06 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247246+0000", 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071939179999999997 2024-04-23T21:21:47.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247321+0000", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5110000000000004e-05 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247520+0000", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000198743 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250307+0000", 2024-04-23T21:21:47.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002786462 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.251964+0000", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001657309 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252019+0000", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4728000000000002e-05 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252060+0000", 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1847000000000001e-05 2024-04-23T21:21:47.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10835 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.021619347, 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011966503999999999, 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10835 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240261+0000", 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240262+0000", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9899999999999999e-07 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240268+0000", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7370000000000003e-06 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240279+0000", 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1569000000000001e-05 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240287+0000", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.926e-06 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247891+0000", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0076038410000000001 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.247988+0000", 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6704000000000001e-05 2024-04-23T21:21:47.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248469+0000", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00048112900000000001 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250313+0000", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018440119999999999 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252122+0000", 2024-04-23T21:21:47.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0018091990000000001 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252178+0000", 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5603000000000003e-05 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252228+0000", 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9985000000000003e-05 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10836 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.021091358, 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011591181000000001, 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10836 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240789+0000", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240790+0000", 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7400000000000002e-07 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240795+0000", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4689999999999999e-06 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240803+0000", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4940000000000003e-06 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.240808+0000", 2024-04-23T21:21:47.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9729999999999997e-06 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.248916+0000", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081076729999999993 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249008+0000", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2650000000000002e-05 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.249420+0000", 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00041150699999999998 2024-04-23T21:21:47.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250324+0000", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00090375899999999996 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252289+0000", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0019649149999999998 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252343+0000", 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3977999999999997e-05 2024-04-23T21:21:47.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252381+0000", 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:10839 3.10 3:0c04bc29:::rbd_object_map.111823663ad5:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.020094099, 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010757071999999999, 2024-04-23T21:21:47.341 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 10839 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241787+0000", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9999999999999997e-07 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241792+0000", 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4460000000000003e-06 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241804+0000", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.257e-05 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.241812+0000", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3089999999999998e-06 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250357+0000", 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0085456630000000002 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250606+0000", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00024869900000000001 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.250982+0000", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00037560399999999998 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252243+0000", 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001261306 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252436+0000", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019354800000000001 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252502+0000", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5513999999999996e-05 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:34.252544+0000", 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1712999999999999e-05 2024-04-23T21:21:47.346 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:18898 3.a 3:5512b404:::rbd_data.11181e0f6ea.0000000000000058:head [truncate 163840] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.6192775230000001, 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063585339999999999, 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 18898 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642603+0000", 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642604+0000", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642607+0000", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.836e-06 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642613+0000", 2024-04-23T21:21:47.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.642618+0000", 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3730000000000001e-06 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643185+0000", 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00056650699999999995 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643263+0000", 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8778999999999999e-05 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.643395+0000", 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00013110699999999999 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.645694+0000", 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0022991880000000002 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648900+0000", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032063669999999999 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648937+0000", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6613999999999999e-05 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.648962+0000", 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4953999999999999e-05 2024-04-23T21:21:47.352 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19031 3.a 3:50c752db:::rbd_data.11181e0f6ea.0000000000000057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4902247380000002, 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0070230910000000004, 2024-04-23T21:21:47.353 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19031 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771656+0000", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5400000000000002e-07 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771658+0000", 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7099999999999999e-06 2024-04-23T21:21:47.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771662+0000", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4630000000000003e-06 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771672+0000", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.115e-06 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772211+0000", 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00053961700000000005 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772288+0000", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6878000000000005e-05 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772401+0000", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011273300000000001 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.773095+0000", 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00069380099999999997 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778585+0000", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0054904460000000004 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778641+0000", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5720000000000002e-05 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778679+0000", 2024-04-23T21:21:47.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8254000000000003e-05 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19032 3.a 3:5512b404:::rbd_data.11181e0f6ea.0000000000000058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4901993689999999, 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0072067279999999999, 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19032 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771681+0000", 2024-04-23T21:21:47.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771682+0000", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771684+0000", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771691+0000", 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8329999999999996e-06 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.771696+0000", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2059999999999998e-06 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772660+0000", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00096427100000000005 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772724+0000", 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3350999999999997e-05 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.772825+0000", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101366 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778690+0000", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058654750000000002 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778781+0000", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0169999999999999e-05 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778839+0000", 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8674000000000001e-05 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778888+0000", 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8779999999999997e-05 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19050 3.a 3:5676f4c4:::rbd_data.11181e0f6ea.000000000000008f:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:47.364 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4830667030000004, 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055626549999999997, 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19050 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:47.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778814+0000", 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778815+0000", 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1829999999999999e-06 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778820+0000", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8910000000000003e-06 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778832+0000", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1499e-05 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778839+0000", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5209999999999997e-06 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778930+0000", 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1051999999999995e-05 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779020+0000", 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9536000000000005e-05 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779126+0000", 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000105986 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.781426+0000", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0023008270000000001 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784299+0000", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028725930000000001 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784351+0000", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2309999999999999e-05 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784377+0000", 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5256999999999999e-05 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:19054 3.a 3:57d3f192:::rbd_data.11181e0f6ea.00000000000000b3:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4829267020000003, 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055262030000000004, 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:47.370 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 19054 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778954+0000", 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778955+0000", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5300000000000003e-07 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778959+0000", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4490000000000001e-06 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778970+0000", 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1161999999999999e-05 2024-04-23T21:21:47.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.778979+0000", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2349999999999999e-06 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779424+0000", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044492400000000002 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779505+0000", 2024-04-23T21:21:47.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1559999999999998e-05 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.779637+0000", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000132395 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784172+0000", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0045345419999999999 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784411+0000", 2024-04-23T21:21:47.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000238994 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784441+0000", 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0261999999999999e-05 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:41.784480+0000", 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8927000000000002e-05 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.375 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.376 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:47.376 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:47.263+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:47.378 INFO:teuthology.orchestra.run.smithi111.stdout:2172 read 0x189e7ef thru 0x18a9cbc (0xb4ce bytes) 2024-04-23T21:21:47.378 INFO:teuthology.orchestra.run.smithi111.stdout:2173 read 0xc8b329 thru 0xc8d005 (0x1cdd bytes) 2024-04-23T21:21:47.378 INFO:teuthology.orchestra.run.smithi111.stdout:2174 write 0x3a373c8 thru 0x3a3ab54 (0x378d bytes) 2024-04-23T21:21:47.378 INFO:teuthology.orchestra.run.smithi111.stdout:2176 clone 19 order 22 su 16384 sc 5 2024-04-23T21:21:47.378 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:47.350+0000 7f4a10b2d640 -1 osd.1 17 log_to_monitors true 2024-04-23T21:21:47.378 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:47.379 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:47.378+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:47.478 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:47.479+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:47.521 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:47.521 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:47.529 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:47.579 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:47.580+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:47.671 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:47.672 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:47.672 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:47.672 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:47.672 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:47.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:47.681 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:47.681+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:47.781 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:47.782+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:47.784 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:47.784 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:47.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3197 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2916352~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364280752999999, 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024247707, 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3197 2024-04-23T21:21:47.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420532+0000", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420534+0000", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.074e-06 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420544+0000", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3370000000000006e-06 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420549+0000", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9030000000000003e-06 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444634+0000", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024085517000000001 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444713+0000", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8632000000000002e-05 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444780+0000", 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6797000000000004e-05 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3198 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2981888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364257145, 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024356404000000002, 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.789 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3198 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420556+0000", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420558+0000", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420564+0000", 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7089999999999999e-06 2024-04-23T21:21:47.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420568+0000", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6280000000000004e-06 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444802+0000", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024234169999999999 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444844+0000", 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-04-23T21:21:47.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444912+0000", 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8367999999999993e-05 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364238191999998, 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.793 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:21:47.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:21:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.797 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364210829000001, 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.801 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364196580000002, 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.802 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364174337000001, 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.806 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:21:47.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:21:47.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.810 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364108851000001, 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364083108999999, 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.818 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364059099999999, 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:21:47.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:21:47.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:21:47.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:21:47.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364036541000001, 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.364016738, 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.827 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:47.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363997252000001, 2024-04-23T21:21:47.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:21:47.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:21:47.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363977453, 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.839 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363957751000001, 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363901625, 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:21:47.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:21:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363864404000001, 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:21:47.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363814090999998, 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:21:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363781196000001, 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:21:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:21:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363758892, 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:21:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:21:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.363738476000002, 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:21:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:47.870 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:47.872 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:47.916 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:47.917+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:47.941 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:47.942+0000 7f4a048e3640 -1 osd.1 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:21:47.982 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:47.984+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:47.987 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:47.987 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:47.987 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:47.987 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:47.997 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:48.010 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone18 from 0x4727eac (overlap 0x4727eac) to 0x1558c66 2024-04-23T21:21:48.084 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:48.084+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:48.136 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:48.136 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:48.136 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:48.136 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:48.136 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:48.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:48.164 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-04-23T21:21:48.166 INFO:teuthology.orchestra.run.smithi111.stdout:2177 read 0x1943ab thru 0x1966cc (0x2322 bytes) 2024-04-23T21:21:48.167 INFO:teuthology.orchestra.run.smithi111.stdout:2178 punch from 0x13ef9d8 to 0x13fb939, (0xbf61 bytes) 2024-04-23T21:21:48.170 INFO:teuthology.orchestra.run.smithi111.stdout:2179 punch from 0x28834f5 to 0x28853d9, (0x1ee4 bytes) 2024-04-23T21:21:48.171 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.171 INFO:teuthology.orchestra.run.smithi111.stdout:2181 punch from 0x3021317 to 0x30229d5, (0x16be bytes) 2024-04-23T21:21:48.171 INFO:teuthology.orchestra.run.smithi111.stdout:2182 writesame 0x1065d1f thru 0x106eee4 data_size 0x2(0x91c6 bytes) 2024-04-23T21:21:48.185 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:48.186+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:48.222 INFO:teuthology.orchestra.run.smithi111.stdout:2183 writesame 0xe405ab thru 0xe4a3b2 data_size 0x34(0x9e08 bytes) 2024-04-23T21:21:48.249 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.257 INFO:teuthology.orchestra.run.smithi111.stdout:2185 write 0x5b51cad thru 0x5b54c88 (0x2fdc bytes) 2024-04-23T21:21:48.260 INFO:teuthology.orchestra.run.smithi111.stdout:2187 punch from 0x67e2b4 to 0x68a406, (0xc152 bytes) 2024-04-23T21:21:48.270 INFO:teuthology.orchestra.run.smithi111.stdout:2189 writesame 0x924c303 thru 0x92585a6 data_size 0x2(0xc2a4 bytes) 2024-04-23T21:21:48.270 INFO:teuthology.orchestra.run.smithi111.stdout:2190 read 0x45fd45f thru 0x46014be (0x4060 bytes) 2024-04-23T21:21:48.270 INFO:teuthology.orchestra.run.smithi111.stdout:2191 write 0x20a3a13 thru 0x20ac604 (0x8bf2 bytes) 2024-04-23T21:21:48.278 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:48.278 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:48.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:48.278 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847591019999999, 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:21:48.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:21:48.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847568208999999, 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:21:48.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847548599, 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.291 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847529735999998, 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.292 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847510441000001, 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.296 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:48.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847483879999999, 2024-04-23T21:21:48.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:21:48.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:21:48.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847442650000001, 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:21:48.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:21:48.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:21:48.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847408923, 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.309 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:21:48.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:21:48.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847385513999999, 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.313 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.317 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847352264000001, 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.321 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847328870999998, 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:21:48.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847308322, 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:21:48.326 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.330 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.847282562, 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:21:48.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.846444546000001, 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.335 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:21:48.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.846421214999999, 2024-04-23T21:21:48.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:21:48.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.845533922000001, 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:21:48.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.845300342000002, 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.348 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.352 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.845183643999999, 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:48.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:21:48.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:21:48.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.844809229999999, 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:21:48.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:21:48.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:21:48.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:21:48.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.843452344999999, 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.364 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.368 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:48.370 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:48.370+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:48.370 INFO:teuthology.orchestra.run.smithi111.stdout:2192 read 0x49596e6 thru 0x495ca3d (0x3358 bytes) 2024-04-23T21:21:48.370 INFO:teuthology.orchestra.run.smithi111.stdout:2193 write 0x75df27c thru 0x75e0e18 (0x1b9d bytes) 2024-04-23T21:21:48.370 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2195 trunc from 0x92585a7 to 0x2acdff4 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2197 read 0x16aff18 thru 0x16ba317 (0xa400 bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2198 punch from 0x1fc4bc to 0x20085f, (0x43a3 bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2199 write 0x9c6b13e thru 0x9c6bd3e (0xc01 bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2200 writesame 0xec9345f thru 0xec98f5d data_size 0xf(0x5aff bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2201 write 0x254c8a6 thru 0x25517a0 (0x4efb bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2202 trunc from 0xec98f5e to 0x7c45e99 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2204 write 0x3b2d9e8 thru 0x3b392f5 (0xb90e bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2205 read 0x1d157b5 thru 0x1d1d99b (0x81e7 bytes) 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.373 INFO:teuthology.orchestra.run.smithi111.stdout:2207 read 0x5e3cf09 thru 0x5e497f4 (0xc8ec bytes) 2024-04-23T21:21:48.374 INFO:teuthology.orchestra.run.smithi111.stdout:2208 read 0x552ac74 thru 0x5539ad4 (0xee61 bytes) 2024-04-23T21:21:48.382 INFO:teuthology.orchestra.run.smithi111.stdout:2209 writesame 0xe4c5e35 thru 0xe4cf942 data_size 0x2(0x9b0e bytes) 2024-04-23T21:21:48.382 INFO:teuthology.orchestra.run.smithi111.stdout:2210 writesame 0x4d82514 thru 0x4d854c1 data_size 0x11(0x2fae bytes) 2024-04-23T21:21:48.385 INFO:teuthology.orchestra.run.smithi111.stdout:2211 writesame 0x9401fb thru 0x948a8b data_size 0x1(0x8891 bytes) 2024-04-23T21:21:48.395 INFO:teuthology.orchestra.run.smithi111.stdout:2212 read 0x2071875 thru 0x20793bb (0x7b47 bytes) 2024-04-23T21:21:48.395 INFO:teuthology.orchestra.run.smithi111.stdout:2213 writesame 0x685b41f thru 0x685d77f data_size 0x1(0x2361 bytes) 2024-04-23T21:21:48.400 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:48.401 INFO:teuthology.orchestra.run.smithi111.stdout:2214 write 0xacf1df9 thru 0xad004e5 (0xe6ed bytes) 2024-04-23T21:21:48.406 INFO:teuthology.orchestra.run.smithi111.stdout:2216 write 0x40a7704 thru 0x40aff1e (0x881b bytes) 2024-04-23T21:21:48.417 INFO:teuthology.orchestra.run.smithi111.stdout:2217 write 0xee112fe thru 0xee19600 (0x8303 bytes) 2024-04-23T21:21:48.427 INFO:teuthology.orchestra.run.smithi111.stdout:2218 write 0x662d207 thru 0x663395a (0x6754 bytes) 2024-04-23T21:21:48.438 INFO:teuthology.orchestra.run.smithi111.stdout:2220 trunc from 0xee19601 to 0x83a873f 2024-04-23T21:21:48.454 INFO:teuthology.orchestra.run.smithi111.stdout:2221 write 0x9e55298 thru 0x9e5d4be (0x8227 bytes) 2024-04-23T21:21:48.458 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.458 INFO:teuthology.orchestra.run.smithi111.stdout:2223 write 0x37d5420 thru 0x37e523c (0xfe1d bytes) 2024-04-23T21:21:48.461 INFO:teuthology.orchestra.run.smithi111.stdout:2224 punch from 0x94210c1 to 0x942762b, (0x656a bytes) 2024-04-23T21:21:48.461 INFO:teuthology.orchestra.run.smithi111.stdout:2225 write 0x76ada6f thru 0x76b2156 (0x46e8 bytes) 2024-04-23T21:21:48.463 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.463 INFO:teuthology.orchestra.run.smithi111.stdout:2229 write 0x84cb9d5 thru 0x84d0553 (0x4b7f bytes) 2024-04-23T21:21:48.469 INFO:teuthology.orchestra.run.smithi111.stdout:2230 read 0x6125fd7 thru 0x612ce61 (0x6e8b bytes) 2024-04-23T21:21:48.469 INFO:teuthology.orchestra.run.smithi111.stdout:2231 write 0x4dbeb7f thru 0x4dc5b30 (0x6fb2 bytes) 2024-04-23T21:21:48.470 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:48.471+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:48.470 INFO:teuthology.orchestra.run.smithi111.stdout:2232 read 0x3ab17e9 thru 0x3abe894 (0xd0ac bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2233 punch from 0x3d9ad72 to 0x3da4ac3, (0x9d51 bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2234 punch from 0x930c3e8 to 0x931038d, (0x3fa5 bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2235 read 0x3b62bc8 thru 0x3b6b940 (0x8d79 bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2237 punch from 0x84a84b6 to 0x84b475a, (0xc2a4 bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2239 read 0x5e2f1e6 thru 0x5e38523 (0x933e bytes) 2024-04-23T21:21:48.472 INFO:teuthology.orchestra.run.smithi111.stdout:2240 write 0x967714c thru 0x967a404 (0x32b9 bytes) 2024-04-23T21:21:48.480 INFO:teuthology.orchestra.run.smithi111.stdout:2241 trunc from 0x9e5d4bf to 0xebbf3c8 2024-04-23T21:21:48.486 INFO:teuthology.orchestra.run.smithi111.stdout:2242 write 0x409c66d thru 0x40aa090 (0xda24 bytes) 2024-04-23T21:21:48.489 INFO:teuthology.orchestra.run.smithi111.stdout:2244 write 0xdf4a293 thru 0xdf59f96 (0xfd04 bytes) 2024-04-23T21:21:48.501 INFO:teuthology.orchestra.run.smithi111.stdout:2245 read 0x4711347 thru 0x471c96b (0xb625 bytes) 2024-04-23T21:21:48.502 INFO:teuthology.orchestra.run.smithi111.stdout:2246 writesame 0x2b99375 thru 0x2b9f423 data_size 0x1(0x60af bytes) 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:47709 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:head [stat out=16b,write-same in=1b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.504206+0000", 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.0030607149999999999, 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0030836969999999998, 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:48.506 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 47709 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504206+0000", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504206+0000", 2024-04-23T21:21:48.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504207+0000", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3399999999999998e-07 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504212+0000", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-06 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504222+0000", 2024-04-23T21:21:48.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0461e-05 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504228+0000", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6489999999999996e-06 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504284+0000", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6369999999999997e-05 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.504372+0000", 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7954000000000005e-05 2024-04-23T21:21:48.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 1 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:48.510 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.511 INFO:teuthology.orchestra.run.smithi111.stdout:2248 write 0xb32308b thru 0xb332f9c (0xff12 bytes) 2024-04-23T21:21:48.514 INFO:teuthology.orchestra.run.smithi111.stdout:2249 write 0x41c98da thru 0x41cb897 (0x1fbe bytes) 2024-04-23T21:21:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:2251 read 0xb7b8304 thru 0xb7c3903 (0xb600 bytes) 2024-04-23T21:21:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:2253 punch from 0x21d094 to 0x221289, (0x41f5 bytes) 2024-04-23T21:21:48.516 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.517 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:48.529 INFO:teuthology.orchestra.run.smithi111.stdout:2257 trunc from 0xebbf3c8 to 0x3637ab2 2024-04-23T21:21:48.544 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:48.545 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:48.545 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-23T21:21:48.559 INFO:teuthology.orchestra.run.smithi111.stdout:2258 write 0x90a2e40 thru 0x90a6672 (0x3833 bytes) 2024-04-23T21:21:48.564 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.565 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:48.565 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:48.571 INFO:teuthology.orchestra.run.smithi111.stdout:2260 write 0xa48cdd4 thru 0xa499134 (0xc361 bytes) 2024-04-23T21:21:48.572 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:48.572+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:48.581 INFO:teuthology.orchestra.run.smithi111.stdout:2261 read 0x6fec62b thru 0x6fee409 (0x1ddf bytes) 2024-04-23T21:21:48.581 INFO:teuthology.orchestra.run.smithi111.stdout:2263 read 0x6fe8956 thru 0x6fee82d (0x5ed8 bytes) 2024-04-23T21:21:48.581 INFO:teuthology.orchestra.run.smithi111.stdout:2264 write 0x5da602b thru 0x5db0150 (0xa126 bytes) 2024-04-23T21:21:48.586 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:48.586 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.591 INFO:teuthology.orchestra.run.smithi111.stdout:2267 read 0x8afc60e thru 0x8b0c5a7 (0xff9a bytes) 2024-04-23T21:21:48.601 INFO:teuthology.orchestra.run.smithi111.stdout:2268 write 0xde7932c thru 0xde8817e (0xee53 bytes) 2024-04-23T21:21:48.606 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.606 INFO:teuthology.orchestra.run.smithi111.stdout:2271 write 0x2d55bea thru 0x2d56af8 (0xf0f bytes) 2024-04-23T21:21:48.607 INFO:teuthology.orchestra.run.smithi111.stdout:2273 read 0xcc19d68 thru 0xcc1db64 (0x3dfd bytes) 2024-04-23T21:21:48.617 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:21:48.618 INFO:teuthology.orchestra.run.smithi111.stdout:2274 trunc from 0xde8817f to 0x4feb256 2024-04-23T21:21:48.620 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:48.620 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:48.630 INFO:teuthology.orchestra.run.smithi111.stdout:2275 read 0x4d2770c thru 0x4d34b9b (0xd490 bytes) 2024-04-23T21:21:48.630 INFO:teuthology.orchestra.run.smithi111.stdout:2276 write 0x39f0f14 thru 0x39fa93a (0x9a27 bytes) 2024-04-23T21:21:48.636 INFO:teuthology.orchestra.run.smithi111.stdout:2277 trunc from 0x4feb256 to 0x95b7e85 2024-04-23T21:21:48.640 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:48.640 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:48.642 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.642 INFO:teuthology.orchestra.run.smithi111.stdout:2281 read 0x6244a77 thru 0x624f9e7 (0xaf71 bytes) 2024-04-23T21:21:48.642 INFO:teuthology.orchestra.run.smithi111.stdout:2282 write 0x46ef348 thru 0x46f1db6 (0x2a6f bytes) 2024-04-23T21:21:48.647 INFO:teuthology.orchestra.run.smithi111.stdout:2285 write 0x8d75ecd thru 0x8d83bd8 (0xdd0c bytes) 2024-04-23T21:21:48.660 INFO:teuthology.orchestra.run.smithi111.stdout:2286 write 0xc195c6a thru 0xc19666f (0xa06 bytes) 2024-04-23T21:21:48.661 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:48.661 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:48.663 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:48.663 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:48.663 INFO:teuthology.orchestra.run.smithi111.stdout:2287 read 0x6bd3dd1 thru 0x6bdb616 (0x7846 bytes) 2024-04-23T21:21:48.663 INFO:teuthology.orchestra.run.smithi111.stdout:2288 read 0x81e0006 thru 0x81e5f66 (0x5f61 bytes) 2024-04-23T21:21:48.670 INFO:teuthology.orchestra.run.smithi111.stdout:2289 trunc from 0xc196670 to 0x6e8f8ae 2024-04-23T21:21:48.673 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:48.674+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:48.676 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:48.680 INFO:teuthology.orchestra.run.smithi111.stdout:2290 write 0x62f53e2 thru 0x6300376 (0xaf95 bytes) 2024-04-23T21:21:48.686 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:48.686 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:21:48.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:21:48.704 INFO:teuthology.orchestra.run.smithi111.stdout:2291 writesame 0xc38ebde thru 0xc391f66 data_size 0x1(0x3389 bytes) 2024-04-23T21:21:48.705 INFO:teuthology.orchestra.run.smithi111.stdout:2294 punch from 0x15791d9 to 0x1587d05, (0xeb2c bytes) 2024-04-23T21:21:48.705 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.711 INFO:teuthology.orchestra.run.smithi111.stdout:2297 writesame 0xdc3f7cf thru 0xdc431f7 data_size 0x7(0x3a29 bytes) 2024-04-23T21:21:48.711 INFO:teuthology.orchestra.run.smithi111.stdout:2298 write 0x9d4798 thru 0x9d5ffb (0x1864 bytes) 2024-04-23T21:21:48.721 INFO:teuthology.orchestra.run.smithi111.stdout:2299 trunc from 0xdc431f8 to 0x74fdf03 2024-04-23T21:21:48.727 INFO:teuthology.orchestra.run.smithi111.stdout:2300 read 0x4c2d831 thru 0x4c2f333 (0x1b03 bytes) 2024-04-23T21:21:48.743 INFO:teuthology.orchestra.run.smithi111.stdout:2301 writesame 0xe6aec63 thru 0xe6bca62 data_size 0x6f(0xde00 bytes) 2024-04-23T21:21:48.748 INFO:teuthology.orchestra.run.smithi111.stdout:2302 trunc from 0xe6bca63 to 0xa84e914 2024-04-23T21:21:48.750 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:48.750 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:48.750 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:48.750 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:2303 read 0x75c1243 thru 0x75cfded (0xebab bytes) 2024-04-23T21:21:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:2304 read 0x1cbb7c thru 0x1d0a0a (0x4e8f bytes) 2024-04-23T21:21:48.754 INFO:teuthology.orchestra.run.smithi111.stdout:2306 read 0x4d0af18 thru 0x4d0e0ab (0x3194 bytes) 2024-04-23T21:21:48.763 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-23T21:21:48.763 INFO:teuthology.orchestra.run.smithi111.stdout:2307 writesame 0xcd6aceb thru 0xcd77ac7 data_size 0xb(0xcddd bytes) 2024-04-23T21:21:48.771 INFO:teuthology.orchestra.run.smithi111.stdout:2309 write 0xeaf551f thru 0xeaf8b9f (0x3681 bytes) 2024-04-23T21:21:48.773 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:48.774+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:48.774 INFO:teuthology.orchestra.run.smithi111.stdout:2310 writesame 0x603767f thru 0x6037d60 data_size 0x1(0x6e2 bytes) 2024-04-23T21:21:48.776 INFO:teuthology.orchestra.run.smithi111.stdout:2311 read 0x89a9c59 thru 0x89b5338 (0xb6e0 bytes) 2024-04-23T21:21:48.776 INFO:teuthology.orchestra.run.smithi111.stdout:2312 write 0x9c3b5c5 thru 0x9c4269f (0x70db bytes) 2024-04-23T21:21:48.779 INFO:teuthology.orchestra.run.smithi111.stdout:2313 write 0x97dc987 thru 0x97ddf7d (0x15f7 bytes) 2024-04-23T21:21:48.781 INFO:teuthology.orchestra.run.smithi111.stdout:2315 read 0x1882dd9 thru 0x188ab06 (0x7d2e bytes) 2024-04-23T21:21:48.781 INFO:teuthology.orchestra.run.smithi111.stdout:2316 writesame 0xceb755e thru 0xcec68df data_size 0x47(0xf382 bytes) 2024-04-23T21:21:48.790 INFO:teuthology.orchestra.run.smithi111.stdout:2317 write 0x1aa5b6d thru 0x1ab534f (0xf7e3 bytes) 2024-04-23T21:21:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:2318 punch from 0x61dfc43 to 0x61ef501, (0xf8be bytes) 2024-04-23T21:21:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.793 INFO:teuthology.orchestra.run.smithi111.stdout:2320 write 0x74b6d5c thru 0x74b848d (0x1732 bytes) 2024-04-23T21:21:48.796 INFO:teuthology.orchestra.run.smithi111.stdout:2321 punch from 0x5f85c7e to 0x5f953a1, (0xf723 bytes) 2024-04-23T21:21:48.803 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:48.803 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:48.803 INFO:teuthology.orchestra.run.smithi111.stdout:2323 trunc from 0xeaf8ba0 to 0x89528c8 2024-04-23T21:21:48.814 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:48.814 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.823 INFO:teuthology.orchestra.run.smithi111.stdout:2325 write 0xbdbe410 thru 0xbdcba59 (0xd64a bytes) 2024-04-23T21:21:48.827 INFO:teuthology.orchestra.run.smithi111.stdout:2326 punch from 0x14b738e to 0x14c69a5, (0xf617 bytes) 2024-04-23T21:21:48.827 INFO:teuthology.orchestra.run.smithi111.stdout:2327 write 0x31bd760 thru 0x31bfcc6 (0x2567 bytes) 2024-04-23T21:21:48.829 INFO:teuthology.orchestra.run.smithi111.stdout:2328 read 0x59b2fbc thru 0x59b5ffe (0x3043 bytes) 2024-04-23T21:21:48.829 INFO:teuthology.orchestra.run.smithi111.stdout:2329 write 0xa46b8da thru 0xa472e27 (0x754e bytes) 2024-04-23T21:21:48.833 INFO:teuthology.orchestra.run.smithi111.stdout:2331 read 0x3ac34e5 thru 0x3ad0068 (0xcb84 bytes) 2024-04-23T21:21:48.833 INFO:teuthology.orchestra.run.smithi111.stdout:2333 read 0x17c5d9d thru 0x17d2fb4 (0xd218 bytes) 2024-04-23T21:21:48.842 INFO:teuthology.orchestra.run.smithi111.stdout:2334 write 0xe8cbf06 thru 0xe8d8439 (0xc534 bytes) 2024-04-23T21:21:48.848 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.849 INFO:teuthology.orchestra.run.smithi111.stdout:2338 write 0x543f46a thru 0x5448bcd (0x9764 bytes) 2024-04-23T21:21:48.850 INFO:teuthology.orchestra.run.smithi111.stdout:2339 punch from 0x3845828 to 0x3846bc7, (0x139f bytes) 2024-04-23T21:21:48.850 INFO:teuthology.orchestra.run.smithi111.stdout:2340 read 0xe3bde0a thru 0xe3bf284 (0x147b bytes) 2024-04-23T21:21:48.851 INFO:teuthology.orchestra.run.smithi111.stdout:2341 write 0x1129424 thru 0x1135ac4 (0xc6a1 bytes) 2024-04-23T21:21:48.852 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.852 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:48.852 INFO:teuthology.orchestra.run.smithi111.stdout:2344 clone 20 order 20 su 65536 sc 14 2024-04-23T21:21:48.874 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:48.874+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:48.935 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:48.935 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:48.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:21:48.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:48.989 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:48.990+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:49.076 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:49.076+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:49.093 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.093 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.102 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:49.176 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:49.177+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:49.205 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.205 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.214 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:49.277 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:49.278+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:49.319 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.319 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:49.378 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:49.378+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:49.430 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.430 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:49.479 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:49.480+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:49.580 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:49.580+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:49.580 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.581 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:49.681 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:49.682+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:49.731 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.731 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:49.784 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:49.785+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:49.843 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.843 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:49.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:49.892 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:49.893+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:49.931 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T21:21:49.936 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:21:49.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-23T21:21:49.983 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:49.984+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:49.998 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:49.998 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.006 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:50.090 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:50.091+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:50.150 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.151 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.159 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:50.200 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:50.201+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:50.285 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:50.286+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:50.302 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.302 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:50.394 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:50.395+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:50.452 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.452 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:50.490 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:50.490+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:50.589 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:50.590+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:50.603 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.603 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.612 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:50.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:50.696 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:50.697+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:50.755 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.755 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:50.791 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:50.792+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:50.812 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:50.812 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:50.829 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:21:50.830 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-04-23T21:21:50.846 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:50.846 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:50.862 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:50.863 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:50.880 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:50.880 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:50.893 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:50.893+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:50.897 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:50.897 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:50.907 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:50.907 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:50.914 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:50.914 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:50.916 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:21:50.931 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:50.932 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:50.933 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T21:21:50.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-23T21:21:50.995 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:50.996+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:51.041 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone19 from 0xe8d843a (overlap 0x2acdff4) to 0xcd7662 2024-04-23T21:21:51.046 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.046 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48241 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:head [stat out=16b,delete] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.0029050199999999999, 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0029321920000000001, 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48241 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045093+0000", 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5689999999999999e-06 2024-04-23T21:21:51.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045097+0000", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.729e-06 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045109+0000", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1652e-05 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045114+0000", 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9899999999999997e-06 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045193+0000", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8844000000000006e-05 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045272+0000", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9672000000000006e-05 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045483+0000", 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021077800000000001 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:48240 3.19 e47/46 3:9b9c2d60:::rbd_data.11183e0a89c0.000000000000000a:head v 47'273, mlcod=24'11)", 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.001938295, 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002011166, 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3600000000000004e-07 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046063+0000", 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0250000000000002e-06 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046074+0000", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046081+0000", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0010000000000004e-06 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046729+0000", 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00064851500000000001 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046749+0000", 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9363000000000001e-05 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 2 2024-04-23T21:21:51.054 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.060 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:21:51.095 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:51.096+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:51.182 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-04-23T21:21:51.185 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.185 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:51.185 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:51.185 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:51.185 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.197 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:21:51.197 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:51.197+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:51.212 INFO:teuthology.orchestra.run.smithi111.stdout:2347 read 0x3443fc thru 0x34d727 (0x932c bytes) 2024-04-23T21:21:51.214 INFO:teuthology.orchestra.run.smithi111.stdout:2348 punch from 0x75ab53c to 0x75b7f18, (0xc9dc bytes) 2024-04-23T21:21:51.217 INFO:teuthology.orchestra.run.smithi111.stdout:2349 write 0xbbcd563 thru 0xbbd45bf (0x705d bytes) 2024-04-23T21:21:51.230 INFO:teuthology.orchestra.run.smithi111.stdout:2350 punch from 0x6639be9 to 0x66453aa, (0xb7c1 bytes) 2024-04-23T21:21:51.230 INFO:teuthology.orchestra.run.smithi111.stdout:2351 punch from 0xcf2a539 to 0xcf3762a, (0xd0f1 bytes) 2024-04-23T21:21:51.238 INFO:teuthology.orchestra.run.smithi111.stdout:2352 trunc from 0xe8d843a to 0x742a21d 2024-04-23T21:21:51.273 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.273 INFO:teuthology.orchestra.run.smithi111.stdout:2355 punch from 0x132e0c1 to 0x133a47f, (0xc3be bytes) 2024-04-23T21:21:51.284 INFO:teuthology.orchestra.run.smithi111.stdout:2356 write 0xc69a5c5 thru 0xc6a21bd (0x7bf9 bytes) 2024-04-23T21:21:51.288 INFO:teuthology.orchestra.run.smithi111.stdout:2357 read 0x5893c83 thru 0x5899f0e (0x628c bytes) 2024-04-23T21:21:51.289 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.289 INFO:teuthology.orchestra.run.smithi111.stdout:2359 write 0x8122988 thru 0x8127848 (0x4ec1 bytes) 2024-04-23T21:21:51.291 INFO:teuthology.orchestra.run.smithi111.stdout:2360 write 0x49287f4 thru 0x4929151 (0x95e bytes) 2024-04-23T21:21:51.298 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:51.298+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:51.299 INFO:teuthology.orchestra.run.smithi111.stdout:2361 write 0x7df353c thru 0x7dfe5b4 (0xb079 bytes) 2024-04-23T21:21:51.299 INFO:teuthology.orchestra.run.smithi111.stdout:2362 write 0x242a322 thru 0x242dba8 (0x3887 bytes) 2024-04-23T21:21:51.307 INFO:teuthology.orchestra.run.smithi111.stdout:2363 write 0xc072638 thru 0xc079bc4 (0x758d bytes) 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:2364 punch from 0xaa280e3 to 0xaa3147e, (0x939b bytes) 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:2366 read 0xa32ea01 thru 0xa32fe40 (0x1440 bytes) 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:2367 read 0x6d690a6 thru 0x6d73950 (0xa8ab bytes) 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.311 INFO:teuthology.orchestra.run.smithi111.stdout:2370 read 0x7c55d6 thru 0x7c678f (0x11ba bytes) 2024-04-23T21:21:51.312 INFO:teuthology.orchestra.run.smithi111.stdout:2371 write 0xaeec6b5 thru 0xaeeeb48 (0x2494 bytes) 2024-04-23T21:21:51.313 INFO:teuthology.orchestra.run.smithi111.stdout:2372 write 0x9db249c thru 0x9db8195 (0x5cfa bytes) 2024-04-23T21:21:51.316 INFO:teuthology.orchestra.run.smithi111.stdout:2373 read 0x4dc2b5f thru 0x4dc808f (0x5531 bytes) 2024-04-23T21:21:51.316 INFO:teuthology.orchestra.run.smithi111.stdout:2374 write 0x5f4cafc thru 0x5f59acb (0xcfd0 bytes) 2024-04-23T21:21:51.332 INFO:teuthology.orchestra.run.smithi111.stdout:2376 write 0xa0a32c9 thru 0xa0b0a36 (0xd76e bytes) 2024-04-23T21:21:51.335 INFO:teuthology.orchestra.run.smithi111.stdout:2377 read 0x643e1f1 thru 0x6443663 (0x5473 bytes) 2024-04-23T21:21:51.336 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.336 INFO:teuthology.orchestra.run.smithi111.stdout:2380 write 0x116917d thru 0x11751e4 (0xc068 bytes) 2024-04-23T21:21:51.340 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 2.3983538169999998, 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:21:51.341 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:21:51.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:21:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:21:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.0 46/45 [PushOp(3:074685b7:::rbd_data.1118a1ef21e9.000000000000002b:head, version: 21'2, data_included: [2533160~20700,3293173~19914], data_size: 40614, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:074685b7:::rbd_data.1118a1ef21e9.000000000000002b:head@21'2, size: 3313087, copy_subset: [(0, 3313087)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3313087, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.941034+0000", 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3981045650000001, 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001350567, 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941034+0000", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941034+0000", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941035+0000", 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6300000000000004e-07 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941089+0000", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3953999999999999e-05 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941113+0000", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3958999999999999e-05 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941134+0000", 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1160000000000001e-05 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941248+0000", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011428 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.941264+0000", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5543999999999999e-05 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.942385+0000", 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0011208069999999999 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.8 46/45 [PushOp(3:13a437a4:::rbd_data.11183c92f5c.0000000000000002:5, version: 24'8, data_included: [0~327680,1081344~23763,1933312~32768], data_size: 384211, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:13a437a4:::rbd_data.11183c92f5c.0000000000000002:5@24'8, size: 2949120, copy_subset: [(0, 2949120)], clone_subset: {}, snapset: 5=[]:{5: [5]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2949120, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.952482+0000", 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3866563789999999, 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0016633819999999999, 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952482+0000", 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952482+0000", 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952483+0000", 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8599999999999998e-07 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952841+0000", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000358415 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952865+0000", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3716000000000001e-05 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952886+0000", 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0443000000000001e-05 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952975+0000", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9191999999999996e-05 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952985+0000", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0392e-05 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.954146+0000", 2024-04-23T21:21:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001160538 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.0 46/45 [PushOp(3:05b80c9a:::rbd_data.1118f7f89011.0000000000000060:6, version: 25'12, data_included: [309713~61120], data_size: 61120, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:05b80c9a:::rbd_data.1118f7f89011.0000000000000060:6@25'12, size: 370833, copy_subset: [(0, 370833)], clone_subset: {}, snapset: 6=[]:{6: [6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 370833, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.952979+0000", 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3861592599999999, 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0011741150000000001, 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952979+0000", 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952979+0000", 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.952980+0000", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0100000000000005e-07 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.953017+0000", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7141000000000002e-05 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.953033+0000", 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5931e-05 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.953043+0000", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8009999999999998e-06 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.953122+0000", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9529000000000004e-05 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.953139+0000", 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6226e-05 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.954154+0000", 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001014886 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.0 46/45 [PushOp(3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12, version: 40'43, data_included: [0~49152,81920~49152,786432~32768,835584~65536,917504~16384,950272~65536,1032192~16384,1703936~262144], data_size: 557056, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12@40'43, size: 3014656, copy_subset: [(0, 3014656)], clone_subset: {}, snapset: 12=[]:{12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3014656, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.969875+0000", 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3692636810000001, 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001875542, 2024-04-23T21:21:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.969875+0000", 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.969875+0000", 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.969876+0000", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7100000000000002e-07 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.970217+0000", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00034137300000000001 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.970240+0000", 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3115000000000001e-05 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.970259+0000", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8865999999999999e-05 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.970342+0000", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3075e-05 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.970357+0000", 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5140000000000001e-05 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.971751+0000", 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001393402 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.8 46/45 [PushOp(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head, version: 44'82, data_included: [0~872402,884736~176174], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head@44'82, size: 4128768, copy_subset: [(0, 4128768)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1060910, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.985891+0000", 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3532475349999999, 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001493215, 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.985891+0000", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.985891+0000", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.985892+0000", 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.986350+0000", 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000458811 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.986365+0000", 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4612e-05 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.986376+0000", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1003e-05 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.986444+0000", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7613000000000005e-05 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.986452+0000", 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7390000000000005e-06 2024-04-23T21:21:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.987384+0000", 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00093203999999999997 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.8 46/45 [PushOp(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head, version: 44'82, data_included: [1060910~36818,1146880~49152,1245184~212992,1507328~65536,1622016~212992,1884160~51852,1982464~212992,2244608~49152,2359296~157090], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head@44'82, size: 4128768, copy_subset: [(0, 4128768)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2516386, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1060910, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.990946+0000", 2024-04-23T21:21:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3481922719999999, 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001815753, 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.990946+0000", 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.990946+0000", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.990947+0000", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.991500+0000", 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00055254200000000005 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.991520+0000", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0278000000000001e-05 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.991548+0000", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.853e-05 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.991619+0000", 2024-04-23T21:21:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0159000000000004e-05 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.991629+0000", 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0149999999999999e-05 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.992762+0000", 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0011334889999999999 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.8 46/45 [PushOp(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head, version: 44'82, data_included: [2516386~39518,2621440~49152,2719744~212992,2981888~49152,3080192~212992,3342336~65536,3431711~8929,3457024~212992,3719168~49152,3817472~148161], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head@44'82, size: 4128768, copy_subset: [(0, 4128768)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3965633, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2516386, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.994611+0000", 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.34452744, 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0016243780000000001, 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.994611+0000", 2024-04-23T21:21:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.994611+0000", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.994612+0000", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6100000000000001e-07 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995088+0000", 2024-04-23T21:21:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000475971 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995103+0000", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5458e-05 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995116+0000", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.259e-05 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995217+0000", 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101655 2024-04-23T21:21:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995226+0000", 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1070000000000005e-06 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.996236+0000", 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001010036 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.6 46/45 [PushOp(3:6613bbd0:::rbd_data.1118dbdf22af.0000000000000050:a, version: 31'27, data_included: [340335~11624], data_size: 11624, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:6613bbd0:::rbd_data.1118dbdf22af.0000000000000050:a@31'27, size: 351959, copy_subset: [(0, 351959)], clone_subset: {}, snapset: a=[]:{a: [a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 351959, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:49.995406+0000", 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.343732529, 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00095221499999999996, 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995406+0000", 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995406+0000", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995407+0000", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8100000000000003e-07 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995441+0000", 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4727000000000001e-05 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995457+0000", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5372000000000001e-05 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995467+0000", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8300000000000008e-06 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995554+0000", 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7062000000000001e-05 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.995570+0000", 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6331999999999999e-05 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.996358+0000", 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078831099999999998 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.6 46/45 [PushOp(3:6128d7c1:::rbd_data.111825873f8b.000000000000000b:12, version: 40'64, data_included: [16384~49152,98304~32768,786432~49152,851968~98304,966656~64064,1703936~131072], data_size: 424512, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:6128d7c1:::rbd_data.111825873f8b.000000000000000b:12@40'64, size: 3014656, copy_subset: [(0, 3014656)], clone_subset: {}, snapset: 12=[]:{12: [12]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3014656, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:50.004243+0000", 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.3348955419999999, 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001223097, 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004243+0000", 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004243+0000", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004244+0000", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8599999999999998e-07 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004458+0000", 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021446099999999999 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004476+0000", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7535999999999999e-05 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004493+0000", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7263000000000001e-05 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004561+0000", 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.7971000000000004e-05 2024-04-23T21:21:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.004575+0000", 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4314999999999999e-05 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.005466+0000", 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00089086500000000002 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.1a 47/46 [PushOp(3:5c1e5a4f:::rbd_data.1118dbdf22af.000000000000000e:a, version: 31'74, data_included: [0~32768,65536~65536,163840~65536,262144~98304,393216~65536], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:5c1e5a4f:::rbd_data.1118dbdf22af.000000000000000e:a@31'74, size: 491520, copy_subset: [(0, 491520)], clone_subset: {}, snapset: a=[]:{a: [a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 491520, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:50.968568+0000", 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.37057052699999998, 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001058364, 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968568+0000", 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968568+0000", 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968569+0000", 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5400000000000001e-07 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968910+0000", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00034165299999999998 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968936+0000", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5982999999999998e-05 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.968958+0000", 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2018000000000002e-05 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.969027+0000", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8183999999999996e-05 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.969033+0000", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8000000000000001e-06 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.969627+0000", 2024-04-23T21:21:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059307200000000004 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.1a 47/46 [PushOp(3:5f125234:::rbd_data.11185f1e06d0.0000000000000009:14, version: 42'138, data_included: [0~262144], data_size: 262144, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:5f125234:::rbd_data.11185f1e06d0.0000000000000009:14@42'138, size: 262144, copy_subset: [(0, 262144)], clone_subset: {}, snapset: 14=[]:{14: [14]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262144, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:50.987071+0000", 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.352067621, 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00094724000000000002, 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987071+0000", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987071+0000", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987071+0000", 2024-04-23T21:21:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987323+0000", 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00025108900000000002 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987336+0000", 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3729e-05 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987346+0000", 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8109999999999997e-06 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987427+0000", 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0797000000000006e-05 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.987451+0000", 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3679000000000001e-05 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:50.988018+0000", 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00056771000000000002 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.4 47/46 [PushOp(3:25d4eef9:::rbd_data.111823663ad5.0000000000000009:b, version: 32'62, data_included: [0~32768,652332~25822,819200~65536,899594~2655,11705042~9197,12512236~9464], data_size: 145442, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(3:25d4eef9:::rbd_data.111823663ad5.0000000000000009:b@32'62, size: 12521700, copy_subset: [(0, 12521700)], clone_subset: {}, snapset: b=[]:{b: [b]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 12521700, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.012128+0000", 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.32701027500000002, 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00099342000000000007, 2024-04-23T21:21:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012128+0000", 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012128+0000", 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012129+0000", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8999999999999996e-07 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012209+0000", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0384999999999997e-05 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012225+0000", 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5509e-05 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012236+0000", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1001e-05 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012299+0000", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3317000000000004e-05 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.012308+0000", 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3350000000000007e-06 2024-04-23T21:21:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.013122+0000", 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00081428299999999996 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48241 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:head [stat out=16b,delete] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.29404770699999999, 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.003993211, 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48241 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045091+0000", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045093+0000", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5689999999999999e-06 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045097+0000", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.729e-06 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045109+0000", 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1652e-05 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045114+0000", 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9899999999999997e-06 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045193+0000", 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8844000000000006e-05 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045272+0000", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9672000000000006e-05 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.045483+0000", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021077800000000001 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.048922+0000", 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0034389730000000001 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.048973+0000", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1091999999999998e-05 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.049031+0000", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7708999999999999e-05 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.049084+0000", 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3202999999999999e-05 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:48240 3.19 e47/46 3:9b9c2d60:::rbd_data.11183e0a89c0.000000000000000a:head v 47'273, mlcod=24'11)", 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.29308098199999999, 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0030953909999999999, 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046058+0000", 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3600000000000004e-07 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046063+0000", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0250000000000002e-06 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046074+0000", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046081+0000", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0010000000000004e-06 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046729+0000", 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00064851500000000001 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.046749+0000", 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9363000000000001e-05 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.049113+0000", 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0023642300000000002 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.049153+0000", 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0241999999999999e-05 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:48256 3.8 e47/45 3:12dda7d6:::rbd_data.11183e0a89c0.000000000000002c:head v 47'84, mlcod=44'81)", 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.052197+0000", 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.28694146100000001, 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001499315, 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052197+0000", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052197+0000", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052198+0000", 2024-04-23T21:21:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999998e-07 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052206+0000", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6639999999999998e-06 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052216+0000", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0142e-05 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052222+0000", 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6510000000000004e-06 2024-04-23T21:21:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052270+0000", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7444999999999997e-05 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.052287+0000", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7076e-05 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053660+0000", 2024-04-23T21:21:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001373642 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053697+0000", 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6053999999999997e-05 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48262 3.6 3:674acce8:::rbd_data.11183e0a89c0.0000000000000038:head [delete] snapc 16=[16] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.053192+0000", 2024-04-23T21:21:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.28594709200000001, 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0021280650000000002, 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48262 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053192+0000", 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053192+0000", 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053194+0000", 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9819999999999998e-06 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053200+0000", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9029999999999998e-06 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053215+0000", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4768e-05 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053223+0000", 2024-04-23T21:21:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0960000000000001e-06 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053287+0000", 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3644000000000003e-05 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053373+0000", 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6199000000000006e-05 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 0", 2024-04-23T21:21:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.053621+0000", 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000247538 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.055072+0000", 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001451323 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.055223+0000", 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015050900000000001 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.055273+0000", 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0674999999999998e-05 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.055320+0000", 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6427999999999997e-05 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:48276 3.8 e47/45 3:15a10495:::rbd_data.11183e0a89c0.0000000000000004:head v 47'86, mlcod=44'82)", 2024-04-23T21:21:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.066093+0000", 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.27304525800000001, 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0026449080000000001, 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066093+0000", 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066093+0000", 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066094+0000", 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1999999999999999e-07 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066101+0000", 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-06 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066112+0000", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1291000000000001e-05 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066120+0000", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8089999999999993e-06 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066217+0000", 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7158999999999996e-05 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.066240+0000", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2878999999999999e-05 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.068705+0000", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0024645549999999998 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.068738+0000", 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3617e-05 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.19 47/46 [PushOp(3:9e93e6b8:::rbd_data.111823663ad5.0000000000000002:head, version: 31'36, data_included: [0~131072,146758~40686,196608~98304,360448~196608,622592~98304,786432~98304,917504~65536,1324287~15471], data_size: 744285, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9e93e6b8:::rbd_data.111823663ad5.0000000000000002:head@31'36, size: 1339758, copy_subset: [(0, 1339758)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1339758, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.077486+0000", 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.26165245999999998, 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0016572920000000001, 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077486+0000", 2024-04-23T21:21:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077486+0000", 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077487+0000", 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4199999999999995e-07 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077950+0000", 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00046311300000000002 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077976+0000", 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5554e-05 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.077998+0000", 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2490999999999999e-05 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.078093+0000", 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4538999999999997e-05 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.078102+0000", 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5089999999999999e-06 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.079144+0000", 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0010414440000000001 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "MOSDPGPush(3.19 47/46 [PushOp(3:9deea9e0:::rbd_data.11188429226.0000000000000007:head, version: 34'94, data_included: [87147~9134,268209~16766,360448~65536,515198~50122,1816471~50075,3634780~56017], data_size: 247650, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:9deea9e0:::rbd_data.11188429226.0000000000000007:head@34'94, size: 3690797, copy_subset: [(0, 3690797)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3690797, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.085718+0000", 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.253420702, 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001214262, 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085718+0000", 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085718+0000", 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085719+0000", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085875+0000", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000156114 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085894+0000", 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9624e-05 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085912+0000", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7847999999999999e-05 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.085996+0000", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4274e-05 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.086010+0000", 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3343000000000001e-05 2024-04-23T21:21:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.086932+0000", 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00092242799999999998 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.428 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.430 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2381 writesame 0xd9e9f2c thru 0xd9f0607 data_size 0x3a(0x66dc bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2382 trunc from 0xd9f0608 to 0xe18dd60 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2383 trunc from 0xe18dd60 to 0xe7f2b42 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2384 write 0xdb922c5 thru 0xdba14c4 (0xf200 bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2386 read 0xc4ef31c thru 0xc4f3a56 (0x473b bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2387 read 0xd89f37b thru 0xd8a49d6 (0x565c bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2388 write 0x1f4424f thru 0x1f4a9f0 (0x67a2 bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2389 writesame 0xdf2b49f thru 0xdf36c5f data_size 0x1(0xb7c1 bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2390 trunc from 0xe7f2b42 to 0x8b02dae 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2391 read 0x67425e0 thru 0x674701c (0x4a3d bytes) 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2392 trunc from 0x8b02dae to 0x9915116 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.432 INFO:teuthology.orchestra.run.smithi111.stdout:2395 write 0x20d97db thru 0x20e8b93 (0xf3b9 bytes) 2024-04-23T21:21:51.433 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:51.430+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:51.439 INFO:teuthology.orchestra.run.smithi111.stdout:2396 writesame 0x6be4f8f thru 0x6be8911 data_size 0x1(0x3983 bytes) 2024-04-23T21:21:51.448 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.448 INFO:teuthology.orchestra.run.smithi111.stdout:2398 read 0x13d38d7 thru 0x13d5db7 (0x24e1 bytes) 2024-04-23T21:21:51.449 INFO:teuthology.orchestra.run.smithi111.stdout:2399 read 0x566e6c0 thru 0x567c719 (0xe05a bytes) 2024-04-23T21:21:51.449 INFO:teuthology.orchestra.run.smithi111.stdout:2401 read 0x54b179e thru 0x54b9f65 (0x87c8 bytes) 2024-04-23T21:21:51.450 INFO:teuthology.orchestra.run.smithi111.stdout:2402 write 0x775cf23 thru 0x7767154 (0xa232 bytes) 2024-04-23T21:21:51.452 INFO:teuthology.orchestra.run.smithi111.stdout:2403 read 0x17fe798 thru 0x18051da (0x6a43 bytes) 2024-04-23T21:21:51.459 INFO:teuthology.orchestra.run.smithi111.stdout:2404 trunc from 0x9915116 to 0x3ee5b85 2024-04-23T21:21:51.516 INFO:teuthology.orchestra.run.smithi111.stdout:2406 write 0x9836fd9 thru 0x983942e (0x2456 bytes) 2024-04-23T21:21:51.519 INFO:teuthology.orchestra.run.smithi111.stdout:2407 read 0x4e12fec thru 0x4e1e6ea (0xb6ff bytes) 2024-04-23T21:21:51.526 INFO:teuthology.orchestra.run.smithi111.stdout:2409 write 0xbb1ede6 thru 0xbb2e42d (0xf648 bytes) 2024-04-23T21:21:51.530 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:51.531+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:51.530 INFO:teuthology.orchestra.run.smithi111.stdout:2411 read 0x74170a5 thru 0x741eaef (0x7a4b bytes) 2024-04-23T21:21:51.530 INFO:teuthology.orchestra.run.smithi111.stdout:2412 read 0x8be2641 thru 0x8bf126e (0xec2e bytes) 2024-04-23T21:21:51.530 INFO:teuthology.orchestra.run.smithi111.stdout:2413 read 0x1e5c080 thru 0x1e5f171 (0x30f2 bytes) 2024-04-23T21:21:51.540 INFO:teuthology.orchestra.run.smithi111.stdout:2414 write 0xeb3363f thru 0xeb3cacd (0x948f bytes) 2024-04-23T21:21:51.542 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.542 INFO:teuthology.orchestra.run.smithi111.stdout:2419 write 0x711a884 thru 0x711acc3 (0x440 bytes) 2024-04-23T21:21:51.553 INFO:teuthology.orchestra.run.smithi111.stdout:2421 trunc from 0xeb3cace to 0x80e9c3c 2024-04-23T21:21:51.562 INFO:teuthology.orchestra.run.smithi111.stdout:2422 punch from 0x1896c8d to 0x1897ff3, (0x1366 bytes) 2024-04-23T21:21:51.563 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.563 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:51.564 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:51.564 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.572 INFO:teuthology.orchestra.run.smithi111.stdout:2423 write 0xb56990a thru 0xb5708ae (0x6fa5 bytes) 2024-04-23T21:21:51.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:51.576 INFO:teuthology.orchestra.run.smithi111.stdout:2424 read 0x5565cc9 thru 0x556a541 (0x4879 bytes) 2024-04-23T21:21:51.576 INFO:teuthology.orchestra.run.smithi111.stdout:2426 write 0x289e9c2 thru 0x28a705b (0x869a bytes) 2024-04-23T21:21:51.584 INFO:teuthology.orchestra.run.smithi111.stdout:2428 writesame 0x90cbdf8 thru 0x90d469d data_size 0x1(0x88a6 bytes) 2024-04-23T21:21:51.604 INFO:teuthology.orchestra.run.smithi111.stdout:2431 writesame 0xe218514 thru 0xe2267cc data_size 0x3(0xe2b9 bytes) 2024-04-23T21:21:51.604 INFO:teuthology.orchestra.run.smithi111.stdout:2432 read 0x91de471 thru 0x91e1d2c (0x38bc bytes) 2024-04-23T21:21:51.604 INFO:teuthology.orchestra.run.smithi111.stdout:2433 read 0x8ef05f4 thru 0x8ef537d (0x4d8a bytes) 2024-04-23T21:21:51.605 INFO:teuthology.orchestra.run.smithi111.stdout:2435 read 0x1b1866d thru 0x1b1abc6 (0x255a bytes) 2024-04-23T21:21:51.605 INFO:teuthology.orchestra.run.smithi111.stdout:2436 read 0x825bcbb thru 0x8268748 (0xca8e bytes) 2024-04-23T21:21:51.605 INFO:teuthology.orchestra.run.smithi111.stdout:2440 read 0xb8daf7e thru 0xb8e04ca (0x554d bytes) 2024-04-23T21:21:51.605 INFO:teuthology.orchestra.run.smithi111.stdout:2441 write 0x5f73665 thru 0x5f74c4c (0x15e8 bytes) 2024-04-23T21:21:51.608 INFO:teuthology.orchestra.run.smithi111.stdout:2442 read 0xccc3d9b thru 0xccc7047 (0x32ad bytes) 2024-04-23T21:21:51.608 INFO:teuthology.orchestra.run.smithi111.stdout:2443 read 0x61311b9 thru 0x61330ba (0x1f02 bytes) 2024-04-23T21:21:51.608 INFO:teuthology.orchestra.run.smithi111.stdout:2444 write 0x165462a thru 0x1657263 (0x2c3a bytes) 2024-04-23T21:21:51.618 INFO:teuthology.orchestra.run.smithi111.stdout:2445 punch from 0x59e9867 to 0x59ea7e6, (0xf7f bytes) 2024-04-23T21:21:51.618 INFO:teuthology.orchestra.run.smithi111.stdout:2446 writesame 0x7a79196 thru 0x7a7d19b data_size 0xb(0x4006 bytes) 2024-04-23T21:21:51.622 INFO:teuthology.orchestra.run.smithi111.stdout:2447 writesame 0x270146 thru 0x273da1 data_size 0x2(0x3c5c bytes) 2024-04-23T21:21:51.631 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:51.632+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:51.637 INFO:teuthology.orchestra.run.smithi111.stdout:2448 read 0xdcf7b18 thru 0xdd04ed5 (0xd3be bytes) 2024-04-23T21:21:51.637 INFO:teuthology.orchestra.run.smithi111.stdout:2449 write 0xc389ddd thru 0xc38cb7a (0x2d9e bytes) 2024-04-23T21:21:51.641 INFO:teuthology.orchestra.run.smithi111.stdout:2450 read 0x1c43e5d thru 0x1c4ea3c (0xabe0 bytes) 2024-04-23T21:21:51.641 INFO:teuthology.orchestra.run.smithi111.stdout:2451 read 0xb1059ee thru 0xb1129d1 (0xcfe4 bytes) 2024-04-23T21:21:51.642 INFO:teuthology.orchestra.run.smithi111.stdout:2452 read 0x183f149 thru 0x184e24c (0xf104 bytes) 2024-04-23T21:21:51.642 INFO:teuthology.orchestra.run.smithi111.stdout:2454 write 0xcccde5f thru 0xccd67bf (0x8961 bytes) 2024-04-23T21:21:51.648 INFO:teuthology.orchestra.run.smithi111.stdout:2455 flatten 2024-04-23T21:21:51.709 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.709 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:51.710 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:51.710 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:51.710 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.721 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:51.764 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:51.765+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:51.832 INFO:teuthology.orchestra.run.smithi111.stdout:2457 writesame 0xd139022 thru 0xd140250 data_size 0x1(0x722f bytes) 2024-04-23T21:21:51.833 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:51.834+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:51.858 INFO:teuthology.orchestra.run.smithi111.stdout:2458 trunc from 0xe2267cd to 0x3ae84ca 2024-04-23T21:21:51.873 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.873 INFO:teuthology.orchestra.run.smithi111.stdout:2461 read 0x17803d thru 0x17e78c (0x6750 bytes) 2024-04-23T21:21:51.879 INFO:teuthology.orchestra.run.smithi111.stdout:2463 trunc from 0x3ae84ca to 0x9633fc8 2024-04-23T21:21:51.886 INFO:teuthology.orchestra.run.smithi111.stdout:2464 read 0x2db4620 thru 0x2dbda2c (0x940d bytes) 2024-04-23T21:21:51.886 INFO:teuthology.orchestra.run.smithi111.stdout:2465 write 0x476a7f thru 0x47b73f (0x4cc1 bytes) 2024-04-23T21:21:51.889 INFO:teuthology.orchestra.run.smithi111.stdout:2466 write 0x5bbdcfe thru 0x5bc7bbb (0x9ebe bytes) 2024-04-23T21:21:51.893 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.893 INFO:teuthology.orchestra.run.smithi111.stdout:2468 write 0x40668 thru 0x40def (0x788 bytes) 2024-04-23T21:21:51.896 INFO:teuthology.orchestra.run.smithi111.stdout:2470 writesame 0xad1971 thru 0xae1670 data_size 0x80(0xfd00 bytes) 2024-04-23T21:21:51.897 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467421740999999, 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:21:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:21:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:21:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467398930000002, 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:21:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:21:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467379319999999, 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467360457000002, 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:21:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:21:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467341162, 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:21:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467314600999998, 2024-04-23T21:21:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467273371000001, 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:21:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:21:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:21:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:21:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467239643999999, 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467216234999999, 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:21:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:21:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467182985000001, 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467159592000002, 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467139043, 2024-04-23T21:21:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.467113283, 2024-04-23T21:21:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:21:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.466275267, 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.466251935999999, 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.465364643000001, 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:21:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:21:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.465131063000001, 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.465014364999998, 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.464639950999999, 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:21:51.973 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.463283065999999, 2024-04-23T21:21:51.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:51.981 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:51.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:21:51.983 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:51.983+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:51.983 INFO:teuthology.orchestra.run.smithi111.stdout:2471 trunc from 0x9633fc8 to 0x79fad23 2024-04-23T21:21:51.983 INFO:teuthology.orchestra.run.smithi111.stdout:2472 write 0x29d21dc thru 0x29dea7c (0xc8a1 bytes) 2024-04-23T21:21:51.984 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:51.985 INFO:teuthology.orchestra.run.smithi111.stdout:2473 write 0x8f11078 thru 0x8f19815 (0x879e bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2474 read 0x3a352b5 thru 0x3a43866 (0xe5b2 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2476 write 0x9c0c7fd thru 0x9c163d2 (0x9bd6 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2478 read 0x35335bc thru 0x35385ab (0x4ff0 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2479 punch from 0x1eaa9d7 to 0x1eb815a, (0xd783 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2481 writesame 0x1a5c06b thru 0x1a642e4 data_size 0x6(0x827a bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2485 write 0x27d975 thru 0x286db7 (0x9443 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2488 punch from 0x31e367f to 0x31eec26, (0xb5a7 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2490 writesame 0x57b9083 thru 0x57beb39 data_size 0x1(0x5ab7 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2491 read 0x377b536 thru 0x3785a15 (0xa4e0 bytes) 2024-04-23T21:21:51.986 INFO:teuthology.orchestra.run.smithi111.stdout:2492 read 0x5e02d40 thru 0x5e075fe (0x48bf bytes) 2024-04-23T21:21:51.987 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T21:21:51.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-23T21:21:52.084 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:52.084+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.131 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:52.131 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:52.131 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.143 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:52.187 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:52.188+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:52.246 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.247 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:52.247 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:52.247 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:52.247 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:52.285 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:52.286+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:52.386 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:52.387+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:52.402 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1713061529999997, 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.171276486, 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:21:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:21:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.171257078, 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:21:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:21:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1712352700000004, 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1712156670000002, 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1711960619999999, 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:21:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:21:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1711752210000004, 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:21:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:21:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:21:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1711459560000002, 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:21:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1711147630000003, 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:21:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:21:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3630317520000004, 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:21:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3598035340000001, 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:21:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3597811530000001, 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3597618750000002, 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3597444410000001, 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3597269909999996, 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:21:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.359709616, 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:21:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3596915740000002, 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.359666453, 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45295 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 53248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3575861180000004, 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010241073999999999, 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.479 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45295 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:52.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044672+0000", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.54e-07 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044677+0000", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8879999999999996e-06 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044688+0000", 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1725e-05 2024-04-23T21:21:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044697+0000", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0340000000000002e-06 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054711+0000", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010013388 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054789+0000", 2024-04-23T21:21:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8527000000000002e-05 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054912+0000", 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012275799999999999 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.3575468810000002, 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:21:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.487 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.489 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:52.489 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:52.489+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:52.589 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:52.590+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:52.592 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:52.592 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:52.592 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:52.690 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:52.691+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:52.702 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.702 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:52.702 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:52.702 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:52.703 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.711 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:52.792 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:52.792+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:52.815 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:52.815 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:52.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:52.815 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:52.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.175663943, 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:21:52.819 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.175631836, 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:21:52.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:21:52.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1756084500000004, 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.824 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:52.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:52.828 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1755881429999997, 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:52.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:21:52.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:21:52.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.175568331, 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:21:52.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:21:52.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1744969330000004, 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.837 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:52.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.841 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1744635880000001, 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1744320589999999, 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:21:52.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:21:52.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:21:52.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:21:52.849 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1744046700000004, 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.850 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743927640000003, 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.854 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.858 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743745199999998, 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:21:52.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743647030000002, 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.863 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:21:52.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743544559999997, 2024-04-23T21:21:52.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:21:52.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743448049999996, 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.874 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:21:52.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:21:52.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1743113059999999, 2024-04-23T21:21:52.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:21:52.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.882 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1706858819999999, 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.883 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.887 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1706680130000002, 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.888 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1706492839999996, 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.892 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:21:52.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:21:52.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1706308559999998, 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.896 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.1706129509999998, 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:21:52.900 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:21:52.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:21:52.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:52.904 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:52.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:52.906 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:52.906+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:52.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:21:52.952 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T21:21:52.962 INFO:teuthology.orchestra.run.smithi111.stdout:2493 write 0xdbb69fc thru 0xdbc2eb8 (0xc4bd bytes) 2024-04-23T21:21:52.977 INFO:teuthology.orchestra.run.smithi111.stdout:2494 trunc from 0xdbc2eb9 to 0x6348a51 2024-04-23T21:21:52.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-23T21:21:52.987 INFO:teuthology.orchestra.run.smithi111.stdout:2495 writesame 0x1a539ce thru 0x1a6032e data_size 0x1(0xc961 bytes) 2024-04-23T21:21:53.006 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:53.007+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:53.022 INFO:teuthology.orchestra.run.smithi111.stdout:2496 trunc from 0x6348a51 to 0xdb69524 2024-04-23T21:21:53.029 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:53.029 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:53.029 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:53.029 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:53.033 INFO:teuthology.orchestra.run.smithi111.stdout:2499 trunc from 0xdb69524 to 0x9489281 2024-04-23T21:21:53.040 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:53.049 INFO:teuthology.orchestra.run.smithi111.stdout:2501 write 0xd978e91 thru 0xd97cec6 (0x4036 bytes) 2024-04-23T21:21:53.052 INFO:teuthology.orchestra.run.smithi111.stdout:2502 clone 21 order 23 su 131072 sc 5 2024-04-23T21:21:53.092 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:53.092 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:53.107 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:53.108+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:53.111 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:21:53.112 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:21:53.128 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:53.129 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:53.145 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:53.145 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:53.162 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:53.162 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:53.179 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:53.184 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.184 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:53.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:53.196 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:53.213 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:53.213 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:53.237 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:53.238+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:53.296 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.297 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:53.309 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:53.310+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:53.410 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:53.411+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:53.448 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.448 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:53.541 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:53.542+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.599 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:53.649 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:53.650+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:53.714 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:53.715+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:53.750 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.751 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:53.814 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:53.815+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:53.862 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:53.862 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:53.870 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:53.954 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:53.955+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:53.961 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T21:21:53.977 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-23T21:21:54.016 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:54.017+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:54.019 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.019 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:54.117 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:54.118+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:54.176 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.176 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.185 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:54.199 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-04-23T21:21:54.218 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:54.219+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:54.225 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.225 INFO:teuthology.orchestra.run.smithi111.stdout:2506 read 0x748f432 thru 0x74998d9 (0xa4a8 bytes) 2024-04-23T21:21:54.234 INFO:teuthology.orchestra.run.smithi111.stdout:2507 trunc from 0xd97cec7 to 0x70fe920 2024-04-23T21:21:54.253 INFO:teuthology.orchestra.run.smithi111.stdout:2509 punch from 0x62d0f72 to 0x62df5c7, (0xe655 bytes) 2024-04-23T21:21:54.253 INFO:teuthology.orchestra.run.smithi111.stdout:2511 read 0x6657573 thru 0x6665317 (0xdda5 bytes) 2024-04-23T21:21:54.254 INFO:teuthology.orchestra.run.smithi111.stdout:2512 punch from 0x9c773d to 0x9c8247, (0xb0a bytes) 2024-04-23T21:21:54.254 INFO:teuthology.orchestra.run.smithi111.stdout:2513 write 0x1f64bba thru 0x1f6fbd1 (0xb018 bytes) 2024-04-23T21:21:54.269 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.269 INFO:teuthology.orchestra.run.smithi111.stdout:2515 read 0x5c4703d thru 0x5c48377 (0x133b bytes) 2024-04-23T21:21:54.280 INFO:teuthology.orchestra.run.smithi111.stdout:2517 write 0xc05311d thru 0xc0586e8 (0x55cc bytes) 2024-04-23T21:21:54.283 INFO:teuthology.orchestra.run.smithi111.stdout:2518 punch from 0xac8f136 to 0xac9c561, (0xd42b bytes) 2024-04-23T21:21:54.283 INFO:teuthology.orchestra.run.smithi111.stdout:2519 read 0x1a04319 thru 0x1a078bb (0x35a3 bytes) 2024-04-23T21:21:54.283 INFO:teuthology.orchestra.run.smithi111.stdout:2521 read 0x164efc1 thru 0x165d047 (0xe087 bytes) 2024-04-23T21:21:54.284 INFO:teuthology.orchestra.run.smithi111.stdout:2522 writesame 0x5aa6024 thru 0x5aaba49 data_size 0xb(0x5a26 bytes) 2024-04-23T21:21:54.286 INFO:teuthology.orchestra.run.smithi111.stdout:2523 punch from 0x446fe76 to 0x4474b04, (0x4c8e bytes) 2024-04-23T21:21:54.286 INFO:teuthology.orchestra.run.smithi111.stdout:2524 writesame 0x5dc9ce8 thru 0x5dd4c2b data_size 0x6(0xaf44 bytes) 2024-04-23T21:21:54.290 INFO:teuthology.orchestra.run.smithi111.stdout:2525 read 0x44551ba thru 0x4461bf0 (0xca37 bytes) 2024-04-23T21:21:54.290 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.290 INFO:teuthology.orchestra.run.smithi111.stdout:2529 read 0x40326aa thru 0x403d0f5 (0xaa4c bytes) 2024-04-23T21:21:54.292 INFO:teuthology.orchestra.run.smithi111.stdout:2530 trunc from 0xc0586e9 to 0xb0bac0a 2024-04-23T21:21:54.299 INFO:teuthology.orchestra.run.smithi111.stdout:2531 write 0x2b78842 thru 0x2b830e2 (0xa8a1 bytes) 2024-04-23T21:21:54.309 INFO:teuthology.orchestra.run.smithi111.stdout:2532 writesame 0xd65f90 thru 0xd68fb5 data_size 0x1(0x3026 bytes) 2024-04-23T21:21:54.319 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:54.321+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:54.329 INFO:teuthology.orchestra.run.smithi111.stdout:2533 read 0x58473b2 thru 0x584b9d4 (0x4623 bytes) 2024-04-23T21:21:54.329 INFO:teuthology.orchestra.run.smithi111.stdout:2534 read 0xab8cb5 thru 0xabe3eb (0x5737 bytes) 2024-04-23T21:21:54.330 INFO:teuthology.orchestra.run.smithi111.stdout:2535 write 0x772341f thru 0x772de84 (0xaa66 bytes) 2024-04-23T21:21:54.336 INFO:teuthology.orchestra.run.smithi111.stdout:2536 trunc from 0xb0bac0a to 0x73d5b6c 2024-04-23T21:21:54.345 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.346 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.347 INFO:teuthology.orchestra.run.smithi111.stdout:2537 write 0x5a7baa1 thru 0x5a88242 (0xc7a2 bytes) 2024-04-23T21:21:54.348 INFO:teuthology.orchestra.run.smithi111.stdout:2538 punch from 0x3f531b8 to 0x3f5f6ed, (0xc535 bytes) 2024-04-23T21:21:54.349 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.349 INFO:teuthology.orchestra.run.smithi111.stdout:2540 read 0x5fcbbfd thru 0x5fd4cf2 (0x90f6 bytes) 2024-04-23T21:21:54.349 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.349 INFO:teuthology.orchestra.run.smithi111.stdout:2542 read 0x57922e2 thru 0x579f40c (0xd12b bytes) 2024-04-23T21:21:54.349 INFO:teuthology.orchestra.run.smithi111.stdout:2543 punch from 0x3c80a8f to 0x3c8b786, (0xacf7 bytes) 2024-04-23T21:21:54.351 INFO:teuthology.orchestra.run.smithi111.stdout:2544 trunc from 0x73d5b6c to 0x662a448 2024-04-23T21:21:54.356 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:54.357 INFO:teuthology.orchestra.run.smithi111.stdout:2545 read 0x1b86095 thru 0x1b93ab7 (0xda23 bytes) 2024-04-23T21:21:54.358 INFO:teuthology.orchestra.run.smithi111.stdout:2546 read 0x1ac47a5 thru 0x1acf512 (0xad6e bytes) 2024-04-23T21:21:54.368 INFO:teuthology.orchestra.run.smithi111.stdout:2548 trunc from 0x662a448 to 0xdc007f8 2024-04-23T21:21:54.380 INFO:teuthology.orchestra.run.smithi111.stdout:2549 writesame 0xdf33ece thru 0xdf384d3 data_size 0x1(0x4606 bytes) 2024-04-23T21:21:54.380 INFO:teuthology.orchestra.run.smithi111.stdout:2551 read 0xc7243d1 thru 0xc730978 (0xc5a8 bytes) 2024-04-23T21:21:54.380 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.380 INFO:teuthology.orchestra.run.smithi111.stdout:2555 read 0xa6bc88f thru 0xa6c1c68 (0x53da bytes) 2024-04-23T21:21:54.380 INFO:teuthology.orchestra.run.smithi111.stdout:2557 writesame 0xc9c9767 thru 0xc9d71bc data_size 0x1(0xda56 bytes) 2024-04-23T21:21:54.405 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.406 INFO:teuthology.orchestra.run.smithi111.stdout:2560 writesame 0x1577a4c thru 0x157fc93 data_size 0x2c(0x8248 bytes) 2024-04-23T21:21:54.407 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.407 INFO:teuthology.orchestra.run.smithi111.stdout:2562 read 0x3484de2 thru 0x348b2ea (0x6509 bytes) 2024-04-23T21:21:54.407 INFO:teuthology.orchestra.run.smithi111.stdout:2563 read 0x6d2e35e thru 0x6d39552 (0xb1f5 bytes) 2024-04-23T21:21:54.414 INFO:teuthology.orchestra.run.smithi111.stdout:2565 trunc from 0xdf384d4 to 0x87c6645 2024-04-23T21:21:54.420 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:54.421+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:54.424 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.424 INFO:teuthology.orchestra.run.smithi111.stdout:2568 read 0x2db4b14 thru 0x2dc096e (0xbe5b bytes) 2024-04-23T21:21:54.425 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.425 INFO:teuthology.orchestra.run.smithi111.stdout:2571 write 0x46046dd thru 0x460da83 (0x93a7 bytes) 2024-04-23T21:21:54.431 INFO:teuthology.orchestra.run.smithi111.stdout:2572 writesame 0x5df4963 thru 0x5df6367 data_size 0x1(0x1a05 bytes) 2024-04-23T21:21:54.435 INFO:teuthology.orchestra.run.smithi111.stdout:2573 writesame 0x54ec40b thru 0x54f7c1b data_size 0x71(0xb811 bytes) 2024-04-23T21:21:54.437 INFO:teuthology.orchestra.run.smithi111.stdout:2574 punch from 0x5fb353 to 0x600088, (0x4d35 bytes) 2024-04-23T21:21:54.437 INFO:teuthology.orchestra.run.smithi111.stdout:2575 write 0x3d43406 thru 0x3d44872 (0x146d bytes) 2024-04-23T21:21:54.438 INFO:teuthology.orchestra.run.smithi111.stdout:2576 read 0x4120a7b thru 0x41298e2 (0x8e68 bytes) 2024-04-23T21:21:54.446 INFO:teuthology.orchestra.run.smithi111.stdout:2577 write 0xacfb293 thru 0xacff5a8 (0x4316 bytes) 2024-04-23T21:21:54.456 INFO:teuthology.orchestra.run.smithi111.stdout:2580 trunc from 0xacff5a9 to 0x4b7dac9 2024-04-23T21:21:54.466 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.467 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.478 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:54.480 INFO:teuthology.orchestra.run.smithi111.stdout:2582 trunc from 0x4b7dac9 to 0x70a33bd 2024-04-23T21:21:54.486 INFO:teuthology.orchestra.run.smithi111.stdout:2583 read 0xd1cc1b thru 0xd1e642 (0x1a28 bytes) 2024-04-23T21:21:54.487 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.487 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.491 INFO:teuthology.orchestra.run.smithi111.stdout:2588 trunc from 0x70a33bd to 0x2e30961 2024-04-23T21:21:54.498 INFO:teuthology.orchestra.run.smithi111.stdout:2589 read 0x2a5c803 thru 0x2a6ab6f (0xe36d bytes) 2024-04-23T21:21:54.499 INFO:teuthology.orchestra.run.smithi111.stdout:2591 trunc from 0x2e30961 to 0x3a9914f 2024-04-23T21:21:54.519 INFO:teuthology.orchestra.run.smithi111.stdout:2592 write 0xb389e47 thru 0xb39806e (0xe228 bytes) 2024-04-23T21:21:54.527 INFO:teuthology.orchestra.run.smithi111.stdout:2593 trunc from 0xb39806f to 0x6ae6486 2024-04-23T21:21:54.537 INFO:teuthology.orchestra.run.smithi111.stdout:2594 writesame 0x5dd2a25 thru 0x5dddbdf data_size 0x1(0xb1bb bytes) 2024-04-23T21:21:54.560 INFO:teuthology.orchestra.run.smithi111.stdout:2596 read 0x1381ab9 thru 0x1387a4c (0x5f94 bytes) 2024-04-23T21:21:54.561 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:54.562+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:54.568 INFO:teuthology.orchestra.run.smithi111.stdout:2597 writesame 0x71daff7 thru 0x71e6865 data_size 0x5f(0xb86f bytes) 2024-04-23T21:21:54.568 INFO:teuthology.orchestra.run.smithi111.stdout:2599 writesame 0x280f20e thru 0x281cd9a data_size 0xf(0xdb8d bytes) 2024-04-23T21:21:54.582 INFO:teuthology.orchestra.run.smithi111.stdout:2602 writesame 0xb58e479 thru 0xb59985d data_size 0x3f(0xb3e5 bytes) 2024-04-23T21:21:54.587 INFO:teuthology.orchestra.run.smithi111.stdout:2604 write 0xbba9f08 thru 0xbbb1d80 (0x7e79 bytes) 2024-04-23T21:21:54.591 INFO:teuthology.orchestra.run.smithi111.stdout:2605 trunc from 0xbbb1d81 to 0xd33cce4 2024-04-23T21:21:54.596 INFO:teuthology.orchestra.run.smithi111.stdout:2606 punch from 0x7af360e to 0x7af5347, (0x1d39 bytes) 2024-04-23T21:21:54.596 INFO:teuthology.orchestra.run.smithi111.stdout:2609 write 0x110104e thru 0x110c0fb (0xb0ae bytes) 2024-04-23T21:21:54.598 INFO:teuthology.orchestra.run.smithi111.stdout:2610 read 0xa30fd66 thru 0xa317420 (0x76bb bytes) 2024-04-23T21:21:54.598 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.600 INFO:teuthology.orchestra.run.smithi111.stdout:2613 trunc from 0xd33cce4 to 0xaf60154 2024-04-23T21:21:54.608 INFO:teuthology.orchestra.run.smithi111.stdout:2614 punch from 0x5ec0820 to 0x5ecddc1, (0xd5a1 bytes) 2024-04-23T21:21:54.608 INFO:teuthology.orchestra.run.smithi111.stdout:2615 punch from 0xa7a1d73 to 0xa7b0442, (0xe6cf bytes) 2024-04-23T21:21:54.608 INFO:teuthology.orchestra.run.smithi111.stdout:2616 read 0x3c7529e thru 0x3c810a9 (0xbe0c bytes) 2024-04-23T21:21:54.608 INFO:teuthology.orchestra.run.smithi111.stdout:2618 punch from 0x548a5eb to 0x5492e87, (0x889c bytes) 2024-04-23T21:21:54.618 INFO:teuthology.orchestra.run.smithi111.stdout:2619 writesame 0xe9d42d8 thru 0xe9db303 data_size 0x6(0x702c bytes) 2024-04-23T21:21:54.619 INFO:teuthology.orchestra.run.smithi111.stdout:2620 punch from 0x91cb17e to 0x91d382e, (0x86b0 bytes) 2024-04-23T21:21:54.619 INFO:teuthology.orchestra.run.smithi111.stdout:2622 writesame 0x96f10ce thru 0x96f6060 data_size 0x1(0x4f93 bytes) 2024-04-23T21:21:54.622 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:54.622+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:54.629 INFO:teuthology.orchestra.run.smithi111.stdout:2626 writesame 0xf08090 thru 0xf16745 data_size 0x1(0xe6b6 bytes) 2024-04-23T21:21:54.633 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.633 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:21:54.680 INFO:teuthology.orchestra.run.smithi111.stdout:2627 trunc from 0xe9db304 to 0x4fb3ba2 2024-04-23T21:21:54.694 INFO:teuthology.orchestra.run.smithi111.stdout:2628 read 0x4bfcf14 thru 0x4c0b20a (0xe2f7 bytes) 2024-04-23T21:21:54.694 INFO:teuthology.orchestra.run.smithi111.stdout:2629 punch from 0x1c394c9 to 0x1c41fa8, (0x8adf bytes) 2024-04-23T21:21:54.695 INFO:teuthology.orchestra.run.smithi111.stdout:2631 read 0x654ac0 thru 0x65f97d (0xaebe bytes) 2024-04-23T21:21:54.695 INFO:teuthology.orchestra.run.smithi111.stdout:2632 read 0x35f3f93 thru 0x35f86f2 (0x4760 bytes) 2024-04-23T21:21:54.695 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.695 INFO:teuthology.orchestra.run.smithi111.stdout:2635 write 0x348b2bb thru 0x3490eaf (0x5bf5 bytes) 2024-04-23T21:21:54.701 INFO:teuthology.orchestra.run.smithi111.stdout:2637 trunc from 0x4fb3ba2 to 0x18a9c9b 2024-04-23T21:21:54.723 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:54.724+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:54.732 INFO:teuthology.orchestra.run.smithi111.stdout:2638 punch from 0xd06a4c to 0xd0fee9, (0x949d bytes) 2024-04-23T21:21:54.752 INFO:teuthology.orchestra.run.smithi111.stdout:2639 write 0xd5c356f thru 0xd5cd1c3 (0x9c55 bytes) 2024-04-23T21:21:54.754 INFO:teuthology.orchestra.run.smithi111.stdout:2640 punch from 0x3cc8120 to 0x3cc84e9, (0x3c9 bytes) 2024-04-23T21:21:54.755 INFO:teuthology.orchestra.run.smithi111.stdout:2641 write 0x6eceb8f thru 0x6ed76af (0x8b21 bytes) 2024-04-23T21:21:54.757 INFO:teuthology.orchestra.run.smithi111.stdout:2642 read 0x941395e thru 0x941cc7e (0x9321 bytes) 2024-04-23T21:21:54.759 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.759 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.767 INFO:teuthology.orchestra.run.smithi111.stdout:2643 trunc from 0xd5cd1c4 to 0x5835b68 2024-04-23T21:21:54.773 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:21:54.780 INFO:teuthology.orchestra.run.smithi111.stdout:2644 trunc from 0x5835b68 to 0x812cbf6 2024-04-23T21:21:54.786 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.786 INFO:teuthology.orchestra.run.smithi111.stdout:2648 writesame 0xdb3114 thru 0xdb90f7 data_size 0x4(0x5fe4 bytes) 2024-04-23T21:21:54.789 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.789 INFO:teuthology.orchestra.run.smithi111.stdout:2650 read 0x623c027 thru 0x6247767 (0xb741 bytes) 2024-04-23T21:21:54.789 INFO:teuthology.orchestra.run.smithi111.stdout:2652 write 0x1f20742 thru 0x1f2156c (0xe2b bytes) 2024-04-23T21:21:54.805 INFO:teuthology.orchestra.run.smithi111.stdout:2653 write 0xe64d9a2 thru 0xe65509c (0x76fb bytes) 2024-04-23T21:21:54.808 INFO:teuthology.orchestra.run.smithi111.stdout:2654 read 0x7e12d82 thru 0x7e1afaa (0x8229 bytes) 2024-04-23T21:21:54.808 INFO:teuthology.orchestra.run.smithi111.stdout:2655 write 0x2ce1380 thru 0x2ce4736 (0x33b7 bytes) 2024-04-23T21:21:54.810 INFO:teuthology.orchestra.run.smithi111.stdout:2656 read 0xb76105a thru 0xb76f4f9 (0xe4a0 bytes) 2024-04-23T21:21:54.811 INFO:teuthology.orchestra.run.smithi111.stdout:2657 trunc from 0xe65509d to 0xdaf075e 2024-04-23T21:21:54.817 INFO:teuthology.orchestra.run.smithi111.stdout:2659 read 0x955cc62 thru 0x9563114 (0x64b3 bytes) 2024-04-23T21:21:54.818 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.818 INFO:teuthology.orchestra.run.smithi111.stdout:2661 writesame 0x54c1741 thru 0x54cc47a data_size 0x26(0xad3a bytes) 2024-04-23T21:21:54.821 INFO:teuthology.orchestra.run.smithi111.stdout:2662 read 0x1075701 thru 0x1077729 (0x2029 bytes) 2024-04-23T21:21:54.824 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:54.825+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:54.832 INFO:teuthology.orchestra.run.smithi111.stdout:2663 trunc from 0xdaf075e to 0x4f74f2d 2024-04-23T21:21:54.843 INFO:teuthology.orchestra.run.smithi111.stdout:2665 read 0x2247fda thru 0x2253d30 (0xbd57 bytes) 2024-04-23T21:21:54.843 INFO:teuthology.orchestra.run.smithi111.stdout:2666 read 0x4ce16 thru 0x4f5ad (0x2798 bytes) 2024-04-23T21:21:54.844 INFO:teuthology.orchestra.run.smithi111.stdout:2669 read 0x1c446da thru 0x1c51c64 (0xd58b bytes) 2024-04-23T21:21:54.844 INFO:teuthology.orchestra.run.smithi111.stdout:2670 write 0x4bf5ad5 thru 0x4bf6a11 (0xf3d bytes) 2024-04-23T21:21:54.847 INFO:teuthology.orchestra.run.smithi111.stdout:2671 punch from 0x10bb380 to 0x10bc18a, (0xe0a bytes) 2024-04-23T21:21:54.861 INFO:teuthology.orchestra.run.smithi111.stdout:2673 write 0xbaba274 thru 0xbabc1cc (0x1f59 bytes) 2024-04-23T21:21:54.864 INFO:teuthology.orchestra.run.smithi111.stdout:2674 read 0xa0473a7 thru 0xa05652d (0xf187 bytes) 2024-04-23T21:21:54.864 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.864 INFO:teuthology.orchestra.run.smithi111.stdout:2676 read 0x57d7e3e thru 0x57e46f8 (0xc8bb bytes) 2024-04-23T21:21:54.864 INFO:teuthology.orchestra.run.smithi111.stdout:2679 writesame 0x57c1198 thru 0x57c7c6c data_size 0x1(0x6ad5 bytes) 2024-04-23T21:21:54.875 INFO:teuthology.orchestra.run.smithi111.stdout:2680 read 0x58ce544 thru 0x58ce777 (0x234 bytes) 2024-04-23T21:21:54.876 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.876 INFO:teuthology.orchestra.run.smithi111.stdout:2684 punch from 0xae78ed4 to 0xae850c6, (0xc1f2 bytes) 2024-04-23T21:21:54.876 INFO:teuthology.orchestra.run.smithi111.stdout:2686 write 0xa8148a9 thru 0xa824745 (0xfe9d bytes) 2024-04-23T21:21:54.879 INFO:teuthology.orchestra.run.smithi111.stdout:2687 read 0x2abf581 thru 0x2ac5353 (0x5dd3 bytes) 2024-04-23T21:21:54.880 INFO:teuthology.orchestra.run.smithi111.stdout:2688 trunc from 0xbabc1cd to 0xc3a08c0 2024-04-23T21:21:54.885 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.890 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:54.890 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:54.891 INFO:teuthology.orchestra.run.smithi111.stdout:2690 write 0xcd5bdea thru 0xcd6a449 (0xe660 bytes) 2024-04-23T21:21:54.895 INFO:teuthology.orchestra.run.smithi111.stdout:2691 punch from 0x7083525 to 0x7089c31, (0x670c bytes) 2024-04-23T21:21:54.895 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.895 INFO:teuthology.orchestra.run.smithi111.stdout:2693 punch from 0x3b9344c to 0x3b9c9a5, (0x9559 bytes) 2024-04-23T21:21:54.895 INFO:teuthology.orchestra.run.smithi111.stdout:2694 trunc from 0xcd6a44a to 0xc4d163b 2024-04-23T21:21:54.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:21:54.906 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.906 INFO:teuthology.orchestra.run.smithi111.stdout:2696 writesame 0x8bce225 thru 0x8bd22b1 data_size 0x5(0x408d bytes) 2024-04-23T21:21:54.910 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.910 INFO:teuthology.orchestra.run.smithi111.stdout:2698 punch from 0x735c09a to 0x736639d, (0xa303 bytes) 2024-04-23T21:21:54.910 INFO:teuthology.orchestra.run.smithi111.stdout:2700 write 0x720af0f thru 0x720cb57 (0x1c49 bytes) 2024-04-23T21:21:54.911 INFO:teuthology.orchestra.run.smithi111.stdout:2701 writesame 0xc1028e5 thru 0xc10f168 data_size 0x52(0xc884 bytes) 2024-04-23T21:21:54.913 INFO:teuthology.orchestra.run.smithi111.stdout:2702 read 0x367aaf1 thru 0x367ced5 (0x23e5 bytes) 2024-04-23T21:21:54.913 INFO:teuthology.orchestra.run.smithi111.stdout:2703 write 0xa98aa25 thru 0xa998045 (0xd621 bytes) 2024-04-23T21:21:54.916 INFO:teuthology.orchestra.run.smithi111.stdout:2705 punch from 0x15ab88a to 0x15b6966, (0xb0dc bytes) 2024-04-23T21:21:54.916 INFO:teuthology.orchestra.run.smithi111.stdout:2708 read 0xb134426 thru 0xb141f8a (0xdb65 bytes) 2024-04-23T21:21:54.923 INFO:teuthology.orchestra.run.smithi111.stdout:2709 trunc from 0xc4d163b to 0x703b411 2024-04-23T21:21:54.925 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:54.926+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:54.933 INFO:teuthology.orchestra.run.smithi111.stdout:2711 read 0x5376259 thru 0x537ed1b (0x8ac3 bytes) 2024-04-23T21:21:54.939 INFO:teuthology.orchestra.run.smithi111.stdout:2712 writesame 0x7bde9e8 thru 0x7be154b data_size 0x2(0x2b64 bytes) 2024-04-23T21:21:54.939 INFO:teuthology.orchestra.run.smithi111.stdout:2713 punch from 0x3c053a6 to 0x3c0da63, (0x86bd bytes) 2024-04-23T21:21:54.948 INFO:teuthology.orchestra.run.smithi111.stdout:2715 write 0xa5128b3 thru 0xa515e0f (0x355d bytes) 2024-04-23T21:21:54.956 INFO:teuthology.orchestra.run.smithi111.stdout:2716 write 0xadbd7d2 thru 0xadbea21 (0x1250 bytes) 2024-04-23T21:21:54.959 INFO:teuthology.orchestra.run.smithi111.stdout:2717 read 0x610488 thru 0x6199cb (0x9544 bytes) 2024-04-23T21:21:54.960 INFO:teuthology.orchestra.run.smithi111.stdout:2718 write 0x6bf6903 thru 0x6bf7bc0 (0x12be bytes) 2024-04-23T21:21:54.961 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T21:21:54.963 INFO:teuthology.orchestra.run.smithi111.stdout:2719 read 0x3b34c8 thru 0x3bee78 (0xb9b1 bytes) 2024-04-23T21:21:54.965 INFO:teuthology.orchestra.run.smithi111.stdout:2720 trunc from 0xadbea22 to 0xb9b8c91 2024-04-23T21:21:54.972 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.972 INFO:teuthology.orchestra.run.smithi111.stdout:2723 trunc from 0xb9b8c91 to 0xb5e16ff 2024-04-23T21:21:54.975 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:21:54.978 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-23T21:21:54.981 INFO:teuthology.orchestra.run.smithi111.stdout:2724 read 0x7fa6a3e thru 0x7fac5c0 (0x5b83 bytes) 2024-04-23T21:21:54.981 INFO:teuthology.orchestra.run.smithi111.stdout:2725 read 0xae65638 thru 0xae66d7e (0x1747 bytes) 2024-04-23T21:21:54.981 INFO:teuthology.orchestra.run.smithi111.stdout:2726 read 0x4940d8c thru 0x494c831 (0xbaa6 bytes) 2024-04-23T21:21:54.988 INFO:teuthology.orchestra.run.smithi111.stdout:2727 writesame 0xc6bdff0 thru 0xc6bef62 data_size 0x7(0xf73 bytes) 2024-04-23T21:21:54.989 INFO:teuthology.orchestra.run.smithi111.stdout:2729 punch from 0x5dda901 to 0x5de2da2, (0x84a1 bytes) 2024-04-23T21:21:54.989 INFO:teuthology.orchestra.run.smithi111.stdout:2730 read 0x2422f09 thru 0x242824c (0x5344 bytes) 2024-04-23T21:21:54.990 INFO:teuthology.orchestra.run.smithi111.stdout:2731 write 0x32ecc70 thru 0x32f8fb2 (0xc343 bytes) 2024-04-23T21:21:54.992 INFO:teuthology.orchestra.run.smithi111.stdout:2732 writesame 0x7ef17cd thru 0x7ef506c data_size 0x30(0x38a0 bytes) 2024-04-23T21:21:54.996 INFO:teuthology.orchestra.run.smithi111.stdout:2733 read 0x3f59c67 thru 0x3f64a8d (0xae27 bytes) 2024-04-23T21:21:54.996 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:54.996 INFO:teuthology.orchestra.run.smithi111.stdout:2737 write 0x227e29b thru 0x227efd5 (0xd3b bytes) 2024-04-23T21:21:55.002 INFO:teuthology.orchestra.run.smithi111.stdout:2738 trunc from 0xc6bef63 to 0x9280bad 2024-04-23T21:21:55.014 INFO:teuthology.orchestra.run.smithi111.stdout:2741 write 0x454d4c8 thru 0x4551df0 (0x4929 bytes) 2024-04-23T21:21:55.016 INFO:teuthology.orchestra.run.smithi111.stdout:2742 write 0x8f2e592 thru 0x8f3cc47 (0xe6b6 bytes) 2024-04-23T21:21:55.018 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:55.018 INFO:teuthology.orchestra.run.smithi111.stdout:2745 trunc from 0x9280bad to 0x8e67f0c 2024-04-23T21:21:55.025 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:55.026+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:55.026 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:55.026 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:55.035 INFO:teuthology.orchestra.run.smithi111.stdout:2746 trunc from 0x8e67f0c to 0x33d5a1d 2024-04-23T21:21:55.039 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:55.046 INFO:teuthology.orchestra.run.smithi111.stdout:2748 read 0x2b6d955 thru 0x2b73335 (0x59e1 bytes) 2024-04-23T21:21:55.046 INFO:teuthology.orchestra.run.smithi111.stdout:2749 read 0x10768a4 thru 0x10802a4 (0x9a01 bytes) 2024-04-23T21:21:55.062 INFO:teuthology.orchestra.run.smithi111.stdout:2750 writesame 0xb337c51 thru 0xb33e760 data_size 0x18(0x6b10 bytes) 2024-04-23T21:21:55.062 INFO:teuthology.orchestra.run.smithi111.stdout:2751 read 0x6e90745 thru 0x6e915fa (0xeb6 bytes) 2024-04-23T21:21:55.063 INFO:teuthology.orchestra.run.smithi111.stdout:2753 read 0x24c40aa thru 0x24c92da (0x5231 bytes) 2024-04-23T21:21:55.065 INFO:teuthology.orchestra.run.smithi111.stdout:2754 trunc from 0xb33e761 to 0xce3933e 2024-04-23T21:21:55.071 INFO:teuthology.orchestra.run.smithi111.stdout:2755 write 0xb46bfca thru 0xb473e71 (0x7ea8 bytes) 2024-04-23T21:21:55.074 INFO:teuthology.orchestra.run.smithi111.stdout:2756 read 0x5975661 thru 0x59784ad (0x2e4d bytes) 2024-04-23T21:21:55.075 INFO:teuthology.orchestra.run.smithi111.stdout:2757 write 0x96b59f4 thru 0x96b7c00 (0x220d bytes) 2024-04-23T21:21:55.077 INFO:teuthology.orchestra.run.smithi111.stdout:2758 read 0xa8fe496 thru 0xa907a33 (0x959e bytes) 2024-04-23T21:21:55.077 INFO:teuthology.orchestra.run.smithi111.stdout:2759 write 0xb06da67 thru 0xb06f685 (0x1c1f bytes) 2024-04-23T21:21:55.082 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:55.082 INFO:teuthology.orchestra.run.smithi111.stdout:2761 write 0x5fc73b4 thru 0x5fd32d3 (0xbf20 bytes) 2024-04-23T21:21:55.085 INFO:teuthology.orchestra.run.smithi111.stdout:2762 write 0x257f8c7 thru 0x2581ddc (0x2516 bytes) 2024-04-23T21:21:55.089 INFO:teuthology.orchestra.run.smithi111.stdout:2763 punch from 0xcc68913 to 0xcc717b3, (0x8ea0 bytes) 2024-04-23T21:21:55.089 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:55.089 INFO:teuthology.orchestra.run.smithi111.stdout:2766 punch from 0x267b1e1 to 0x267f098, (0x3eb7 bytes) 2024-04-23T21:21:55.089 INFO:teuthology.orchestra.run.smithi111.stdout:2768 writesame 0x8295504 thru 0x829c1ec data_size 0x7(0x6ce9 bytes) 2024-04-23T21:21:55.092 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:55.100 INFO:teuthology.orchestra.run.smithi111.stdout:2770 write 0xda558d9 thru 0xda60d5a (0xb482 bytes) 2024-04-23T21:21:55.105 INFO:teuthology.orchestra.run.smithi111.stdout:2772 write 0x4ec0e1d thru 0x4ecb272 (0xa456 bytes) 2024-04-23T21:21:55.109 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:55.109 INFO:teuthology.orchestra.run.smithi111.stdout:2778 write 0x4f6e201 thru 0x4f74da2 (0x6ba2 bytes) 2024-04-23T21:21:55.110 INFO:teuthology.orchestra.run.smithi111.stdout:2779 write 0xac2cc3c thru 0xac36ad4 (0x9e99 bytes) 2024-04-23T21:21:55.113 INFO:teuthology.orchestra.run.smithi111.stdout:2780 clone 22 order 24 su 524288 sc 7 2024-04-23T21:21:55.127 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:55.128+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:55.185 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:55.186 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:55.195 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:55.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:55.228 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:55.229+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:55.330 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:55.331+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:55.341 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:55.342 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:55.351 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:55.356 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:55.356 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:55.375 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:21:55.375 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:21:55.392 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:55.392 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:55.410 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:55.411 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:55.428 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:55.428 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:55.430 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:55.432+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:55.445 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:55.446 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.458 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027567435999998, 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:21:55.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:21:55.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.463 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027544625000001, 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.464 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:21:55.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027525014999998, 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.469 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:21:55.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.473 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027506152000001, 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:21:55.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:21:55.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.478 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027486857, 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:55.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:21:55.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027460296000001, 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:21:55.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027419066, 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.489 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:21:55.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027385338999999, 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.494 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:21:55.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027361930000001, 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:21:55.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.02732868, 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:21:55.504 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:55.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:21:55.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:21:55.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:55.509 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027305287000001, 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:21:55.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.514 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027284737999999, 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.515 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:21:55.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.027258977999999, 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:21:55.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:21:55.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.026420962, 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:21:55.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:21:55.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.026397630999998, 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:21:55.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.025510338, 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.535 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:21:55.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.025276758, 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:21:55.540 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:21:55.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.545 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.025160060000001, 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:21:55.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.024785646000002, 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:21:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.023428761000002, 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:21:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:21:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:55.561 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:55.562 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:55.562 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:55.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:55.564 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:55.565 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:55.565 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:55.563+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:55.663 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:55.664+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:55.706 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:55.707 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:55.707 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:55.707 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:55.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:55.764 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:55.764+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:55.858 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:55.858 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:55.858 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:55.858 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:55.858 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:55.864 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:55.865+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:55.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:55.965 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:55.966+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:55.970 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:21:55.970 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T21:21:55.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7858299609999992, 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.017 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7858002939999995, 2024-04-23T21:21:56.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.025 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7857808859999995, 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:21:56.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7857590779999999, 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:21:56.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:21:56.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:21:56.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7857394749999997, 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.034 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7857198699999994, 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.038 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:56.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:21:56.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:21:56.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7856990289999999, 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.042 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.046 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7856697639999997, 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:56.050 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.7856385709999998, 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.054 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9775555599999999, 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.058 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9743273420000005, 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.062 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9743049609999996, 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:21:56.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:21:56.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:21:56.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:21:56.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9742856829999997, 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.067 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9742682489999996, 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.071 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:21:56.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:21:56.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.974250799, 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.075 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9742334240000003, 2024-04-23T21:21:56.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:21:56.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:21:56.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.083 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9742153819999997, 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:21:56.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9741902610000004, 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45295 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 53248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9721099259999999, 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010241073999999999, 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45295 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044671+0000", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044672+0000", 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.54e-07 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044677+0000", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8879999999999996e-06 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044688+0000", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1725e-05 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044697+0000", 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0340000000000002e-06 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054711+0000", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010013388 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054789+0000", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8527000000000002e-05 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054912+0000", 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012275799999999999 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.095 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9720706889999997, 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.096 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:21:56.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:21:56.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:21:56.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.100 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:56.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:56.102 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone21 from 0xda60d5b (overlap 0x18a9c9b) to 0x163268b 2024-04-23T21:21:56.102 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:56.102+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:56.109 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-04-23T21:21:56.202 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:56.203+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:56.229 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:56.229 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:56.229 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:21:56.229 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:56.239 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:56.324 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:56.325+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:56.361 INFO:teuthology.orchestra.run.smithi111.stdout:2781 write 0x1081a63 thru 0x108a56c (0x8b0a bytes) 2024-04-23T21:21:56.376 INFO:teuthology.orchestra.run.smithi111.stdout:2782 punch from 0xc026eb to 0xc0e949, (0xc25e bytes) 2024-04-23T21:21:56.377 INFO:teuthology.orchestra.run.smithi111.stdout:2784 writesame 0xbe5a02d thru 0xbe5e33c data_size 0x4a(0x4310 bytes) 2024-04-23T21:21:56.384 INFO:teuthology.orchestra.run.smithi111.stdout:2787 write 0x48713e8 thru 0x4879256 (0x7e6f bytes) 2024-04-23T21:21:56.389 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:56.389 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:21:56.389 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:21:56.389 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:21:56.389 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:56.400 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:21:56.407 INFO:teuthology.orchestra.run.smithi111.stdout:2789 trunc from 0xda60d5b to 0x6e5be73 2024-04-23T21:21:56.407 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:56.408+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:56.438 INFO:teuthology.orchestra.run.smithi111.stdout:2790 read 0x3cac820 thru 0x3cb8771 (0xbf52 bytes) 2024-04-23T21:21:56.439 INFO:teuthology.orchestra.run.smithi111.stdout:2792 read 0x5084bb7 thru 0x508868c (0x3ad6 bytes) 2024-04-23T21:21:56.444 INFO:teuthology.orchestra.run.smithi111.stdout:2794 writesame 0x6fd830f thru 0x6fe7804 data_size 0x5(0xf4f6 bytes) 2024-04-23T21:21:56.445 INFO:teuthology.orchestra.run.smithi111.stdout:2795 punch from 0x2c882b1 to 0x2c92182, (0x9ed1 bytes) 2024-04-23T21:21:56.446 INFO:teuthology.orchestra.run.smithi111.stdout:2797 punch from 0x42dd364 to 0x42e27fc, (0x5498 bytes) 2024-04-23T21:21:56.446 INFO:teuthology.orchestra.run.smithi111.stdout:2798 punch from 0x57b0bf9 to 0x57baeb6, (0xa2bd bytes) 2024-04-23T21:21:56.458 INFO:teuthology.orchestra.run.smithi111.stdout:2799 write 0xceca9f8 thru 0xcecfd27 (0x5330 bytes) 2024-04-23T21:21:56.461 INFO:teuthology.orchestra.run.smithi111.stdout:2800 punch from 0xb5ebb03 to 0xb5f32ef, (0x77ec bytes) 2024-04-23T21:21:56.461 INFO:teuthology.orchestra.run.smithi111.stdout:2802 punch from 0x6aa1242 to 0x6aa6400, (0x51be bytes) 2024-04-23T21:21:56.461 INFO:teuthology.orchestra.run.smithi111.stdout:2804 write 0xbe21697 thru 0xbe2e62d (0xcf97 bytes) 2024-04-23T21:21:56.465 INFO:teuthology.orchestra.run.smithi111.stdout:2805 write 0x410c30 thru 0x41644d (0x581e bytes) 2024-04-23T21:21:56.480 INFO:teuthology.orchestra.run.smithi111.stdout:2807 read 0x341424a thru 0x3415d07 (0x1abe bytes) 2024-04-23T21:21:56.480 INFO:teuthology.orchestra.run.smithi111.stdout:2808 read 0x3c90f02 thru 0x3c9b4c3 (0xa5c2 bytes) 2024-04-23T21:21:56.495 INFO:teuthology.orchestra.run.smithi111.stdout:2809 trunc from 0xcecfd28 to 0x16f9ef1 2024-04-23T21:21:56.505 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:56.505+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8858946119999995, 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.526 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.530 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8858625050000004, 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.534 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8858391189999999, 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8858188120000001, 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.539 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:21:56.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8857990000000004, 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:21:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8847276019999999, 2024-04-23T21:21:56.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:56.551 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8846942569999996, 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:21:56.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.555 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8846627280000003, 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:21:56.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:21:56.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8846353390000008, 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.561 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8846234329999998, 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.565 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:56.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:21:56.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:21:56.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8846051890000002, 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.569 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:21:56.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8845953719999997, 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:21:56.573 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.577 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8845851249999992, 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.581 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.884575474, 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:21:56.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:21:56.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:21:56.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:21:56.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8845419749999994, 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.586 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8809165510000003, 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.590 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:56.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:21:56.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:21:56.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8808986819999998, 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:21:56.598 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8808799530000009, 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.599 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8808615250000003, 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:21:56.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.8808436200000003, 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:21:56.607 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:21:56.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:21:56.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:21:56.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:21:56.611 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:21:56.612 INFO:teuthology.orchestra.run.smithi111.stdout:2810 punch from 0x227425 to 0x22a1ad, (0x2d88 bytes) 2024-04-23T21:21:56.612 INFO:teuthology.orchestra.run.smithi111.stdout:2811 punch from 0x16046c0 to 0x160df6f, (0x98af bytes) 2024-04-23T21:21:56.613 INFO:teuthology.orchestra.run.smithi111.stdout:2812 read 0x16a8cb7 thru 0x16afbec (0x6f36 bytes) 2024-04-23T21:21:56.614 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:21:56.614 INFO:teuthology.orchestra.run.smithi111.stdout:2816 writesame 0x5b6c468 thru 0x5b719d3 data_size 0x4d(0x556c bytes) 2024-04-23T21:21:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:2820 writesame 0x18c4880 thru 0x18c7ff9 data_size 0x1b(0x377a bytes) 2024-04-23T21:21:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:2821 read 0x2787fec thru 0x2788481 (0x496 bytes) 2024-04-23T21:21:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:2823 write 0x970e8ed thru 0x9716210 (0x7924 bytes) 2024-04-23T21:21:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:2824 read 0x4c60e81 thru 0x4c63cfe (0x2e7e bytes) 2024-04-23T21:21:56.615 INFO:teuthology.orchestra.run.smithi111.stdout:2826 clone 23 order 22 su 524288 sc 11 2024-04-23T21:21:56.615 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:56.614+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:56.713 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:56.715+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:56.717 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:56.718 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:56.718 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:56.718 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:56.726 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:21:56.814 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:56.815+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:56.828 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:56.829 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:56.829 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:56.829 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:56.829 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:56.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:21:56.915 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:56.916+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:56.971 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026475649, 2024-04-23T21:21:56.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:56.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:21:56.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:21:56.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:56.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:21:56.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026437529000001, 2024-04-23T21:21:56.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:56.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:56.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.02641513, 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:56.994 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:21:56.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026395733999999, 2024-04-23T21:21:56.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:56.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:21:57.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:21:57.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026376136, 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026355923000001, 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:21:57.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026323757, 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:21:57.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:21:57.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026299143999999, 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026278145999999, 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:21:57.023 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026257847, 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:21:57.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.026180058, 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.024684684, 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:21:57.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:21:57.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.024634794000001, 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.036 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.024611025, 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:21:57.040 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:21:57.044 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.024590019, 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:21:57.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.048 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.024570413999999, 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:21:57.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:21:57.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:21:57.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.008568264999999, 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.008549754000001, 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.008529309, 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:21:57.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:21:57.064 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.0294520970000001, 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.065 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:21:57.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:21:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.070 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:57.072 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-23T21:21:57.072 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:57.073 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone22 from 0x9716211 (overlap 0x16f9ef1) to 0x0 2024-04-23T21:21:57.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:21:57.073 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:57.072+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:57.126 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-04-23T21:21:57.172 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:57.173+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:57.175 INFO:teuthology.orchestra.run.smithi111.stdout:2828 writesame 0xe478968 thru 0xe48861e data_size 0x5(0xfcb7 bytes) 2024-04-23T21:21:57.175 INFO:teuthology.orchestra.run.smithi111.stdout:2829 write 0x1d2b2d1 thru 0x1d31023 (0x5d53 bytes) 2024-04-23T21:21:57.179 INFO:teuthology.orchestra.run.smithi111.stdout:2830 read 0xacb7c60 thru 0xacc25b5 (0xa956 bytes) 2024-04-23T21:21:57.179 INFO:teuthology.orchestra.run.smithi111.stdout:2831 read 0x46648be thru 0x466670b (0x1e4e bytes) 2024-04-23T21:21:57.179 INFO:teuthology.orchestra.run.smithi111.stdout:2833 write 0x36ee711 thru 0x36f4d66 (0x6656 bytes) 2024-04-23T21:21:57.183 INFO:teuthology.orchestra.run.smithi111.stdout:2834 punch from 0x590e048 to 0x591ca31, (0xe9e9 bytes) 2024-04-23T21:21:57.183 INFO:teuthology.orchestra.run.smithi111.stdout:2835 write 0x11f59b9 thru 0x1202bb5 (0xd1fd bytes) 2024-04-23T21:21:57.188 INFO:teuthology.orchestra.run.smithi111.stdout:2836 write 0x62ab21f thru 0x62b28a2 (0x7684 bytes) 2024-04-23T21:21:57.191 INFO:teuthology.orchestra.run.smithi111.stdout:2838 read 0x5ccd475 thru 0x5cd4d78 (0x7904 bytes) 2024-04-23T21:21:57.191 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:57.191 INFO:teuthology.orchestra.run.smithi111.stdout:2841 write 0xc5d51fa thru 0xc5db33f (0x6146 bytes) 2024-04-23T21:21:57.194 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:57.206 INFO:teuthology.orchestra.run.smithi111.stdout:2844 trunc from 0xe48861f to 0x43fa35b 2024-04-23T21:21:57.227 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:57.227 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:57.227 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:57.227 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:57.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:21:57.241 INFO:teuthology.orchestra.run.smithi111.stdout:2846 writesame 0x77fc284 thru 0x780a0c4 data_size 0x1(0xde41 bytes) 2024-04-23T21:21:57.241 INFO:teuthology.orchestra.run.smithi111.stdout:2849 write 0x76636a4 thru 0x7663aa6 (0x403 bytes) 2024-04-23T21:21:57.255 INFO:teuthology.orchestra.run.smithi111.stdout:2850 write 0xd430bfd thru 0xd438180 (0x7584 bytes) 2024-04-23T21:21:57.259 INFO:teuthology.orchestra.run.smithi111.stdout:2852 read 0xb4eb6b8 thru 0xb4f0f9d (0x58e6 bytes) 2024-04-23T21:21:57.259 INFO:teuthology.orchestra.run.smithi111.stdout:2854 write 0x7432e9e thru 0x7433991 (0xaf4 bytes) 2024-04-23T21:21:57.261 INFO:teuthology.orchestra.run.smithi111.stdout:2856 read 0x26200e5 thru 0x2627128 (0x7044 bytes) 2024-04-23T21:21:57.262 INFO:teuthology.orchestra.run.smithi111.stdout:2857 read 0xbcd3f8a thru 0xbcd52d4 (0x134b bytes) 2024-04-23T21:21:57.271 INFO:teuthology.orchestra.run.smithi111.stdout:2859 trunc from 0xd438181 to 0x60eb2a9 2024-04-23T21:21:57.273 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:57.274+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:57.281 INFO:teuthology.orchestra.run.smithi111.stdout:2860 writesame 0x26a7e87 thru 0x26aa77b data_size 0xf(0x28f5 bytes) 2024-04-23T21:21:57.294 INFO:teuthology.orchestra.run.smithi111.stdout:2862 trunc from 0x60eb2a9 to 0xb3c94fd 2024-04-23T21:21:57.300 INFO:teuthology.orchestra.run.smithi111.stdout:2863 read 0xa5c66be thru 0xa5cd4b9 (0x6dfc bytes) 2024-04-23T21:21:57.300 INFO:teuthology.orchestra.run.smithi111.stdout:2865 read 0xadd375a thru 0xadd9bc4 (0x646b bytes) 2024-04-23T21:21:57.300 INFO:teuthology.orchestra.run.smithi111.stdout:2866 write 0x61e070d thru 0x61e3dc4 (0x36b8 bytes) 2024-04-23T21:21:57.303 INFO:teuthology.orchestra.run.smithi111.stdout:2867 punch from 0x5a2050b to 0x5a2d4ab, (0xcfa0 bytes) 2024-04-23T21:21:57.303 INFO:teuthology.orchestra.run.smithi111.stdout:2873 read 0x2a0ef4f thru 0x2a10797 (0x1849 bytes) 2024-04-23T21:21:57.311 INFO:teuthology.orchestra.run.smithi111.stdout:2874 write 0xd6e4638 thru 0xd6f279b (0xe164 bytes) 2024-04-23T21:21:57.314 INFO:teuthology.orchestra.run.smithi111.stdout:2875 write 0xc731c86 thru 0xc73d8fd (0xbc78 bytes) 2024-04-23T21:21:57.318 INFO:teuthology.orchestra.run.smithi111.stdout:2876 punch from 0xad0446d to 0xad0e489, (0xa01c bytes) 2024-04-23T21:21:57.319 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:57.319 INFO:teuthology.orchestra.run.smithi111.stdout:2879 writesame 0x4bc6077 thru 0x4bc7081 data_size 0x3(0x100b bytes) 2024-04-23T21:21:57.319 INFO:teuthology.orchestra.run.smithi111.stdout:2880 write 0x2c086f9 thru 0x2c1301e (0xa926 bytes) 2024-04-23T21:21:57.321 INFO:teuthology.orchestra.run.smithi111.stdout:2881 write 0x6a8a174 thru 0x6a93ba1 (0x9a2e bytes) 2024-04-23T21:21:57.331 INFO:teuthology.orchestra.run.smithi111.stdout:2882 writesame 0xe447afe thru 0xe455c6a data_size 0x1(0xe16d bytes) 2024-04-23T21:21:57.331 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:57.331 INFO:teuthology.orchestra.run.smithi111.stdout:2888 punch from 0xbfb9985 to 0xbfba74a, (0xdc5 bytes) 2024-04-23T21:21:57.331 INFO:teuthology.orchestra.run.smithi111.stdout:2889 write 0x622ccaf thru 0x622e1fc (0x154e bytes) 2024-04-23T21:21:57.333 INFO:teuthology.orchestra.run.smithi111.stdout:2890 read 0x5730e63 thru 0x573185e (0x9fc bytes) 2024-04-23T21:21:57.334 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:57.334 INFO:teuthology.orchestra.run.smithi111.stdout:2892 write 0x4a7b9e0 thru 0x4a800c9 (0x46ea bytes) 2024-04-23T21:21:57.344 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:57.344 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:57.345 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:21:57.345 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:21:57.345 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:57.346 INFO:teuthology.orchestra.run.smithi111.stdout:2893 trunc from 0xe455c6b to 0x4c7d34f 2024-04-23T21:21:57.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:21:57.373 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:57.374+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:57.373 INFO:teuthology.orchestra.run.smithi111.stdout:2894 write 0xc402240 thru 0xc403fb2 (0x1d73 bytes) 2024-04-23T21:21:57.379 INFO:teuthology.orchestra.run.smithi111.stdout:2895 trunc from 0xc403fb3 to 0x9e674b4 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2896 read 0x9e661b7 thru 0x9e674b3 (0x12fd bytes) 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2897 punch from 0x34ccc02 to 0x34d8960, (0xbd5e bytes) 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2899 read 0x681ce49 thru 0x682536d (0x8525 bytes) 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2900 punch from 0x5a7b619 to 0x5a836cf, (0x80b6 bytes) 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2901 read 0x4ff8744 thru 0x500647b (0xdd38 bytes) 2024-04-23T21:21:57.389 INFO:teuthology.orchestra.run.smithi111.stdout:2903 write 0x343ff93 thru 0x344d348 (0xd3b6 bytes) 2024-04-23T21:21:57.390 INFO:teuthology.orchestra.run.smithi111.stdout:2904 write 0x86eb4c6 thru 0x86fa5b2 (0xf0ed bytes) 2024-04-23T21:21:57.394 INFO:teuthology.orchestra.run.smithi111.stdout:2905 writesame 0x4c49028 thru 0x4c54c5a data_size 0x1(0xbc33 bytes) 2024-04-23T21:21:57.416 INFO:teuthology.orchestra.run.smithi111.stdout:2906 write 0x60c97ce thru 0x60d22b4 (0x8ae7 bytes) 2024-04-23T21:21:57.420 INFO:teuthology.orchestra.run.smithi111.stdout:2907 read 0x98c5ae8 thru 0x98cae90 (0x53a9 bytes) 2024-04-23T21:21:57.420 INFO:teuthology.orchestra.run.smithi111.stdout:2910 punch from 0x960a1bc to 0x9616280, (0xc0c4 bytes) 2024-04-23T21:21:57.420 INFO:teuthology.orchestra.run.smithi111.stdout:2911 write 0x772d59b thru 0x773748a (0x9ef0 bytes) 2024-04-23T21:21:57.426 INFO:teuthology.orchestra.run.smithi111.stdout:2912 trunc from 0x9e674b4 to 0x6ebbb33 2024-04-23T21:21:57.432 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:57.433 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:57.446 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-04-23T21:21:56.975919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":880326282,"num_objects":619,"num_object_clones":153,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":144,"num_read":70192,"num_read_kb":270985,"num_write":8018,"num_write_kb":144940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6255,"ondisk_log_size":6255,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297488,"kb_used_data":25860,"kb_used_omap":17,"kb_used_meta":271470,"kb_avail":754677232,"statfs":{"total":773094113280,"available":772789485568,"internally_reserved":0,"allocated":26480640,"data_stored":138249589,"data_compressed":7273840,"data_compressed_allocated":8871936,"data_compressed_original":125170848,"omap_allocated":17904,"internal_metadata":277985808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63113314,"num_objects":87,"num_object_clones":33,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":28,"num_read":6368,"num_read_kb":57347,"num_write":645,"num_write_kb":15607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":4328802,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.022062"},"pg_stats":[{"pgid":"3.1f","version":"47'202","reported_seq":2430,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983371+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:21:52.983371+0000","last_peered":"2024-04-23T21:21:52.983371+0000","last_clean":"2024-04-23T21:21:52.983371+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:21:52.983371+0000","last_undegraded":"2024-04-23T21:21:52.983371+0000","last_fullsized":"2024-04-23T21:21:52.983371+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:40:03.233990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":35796609,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":2638,"num_read_kb":4595,"num_write":274,"num_write_kb":2091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"47'142","reported_seq":3272,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955990+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:21:52.955990+0000","last_peered":"2024-04-23T21:21:52.955990+0000","last_clean":"2024-04-23T21:21:52.955990+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:21:52.955990+0000","last_undegraded":"2024-04-23T21:21:52.955990+0000","last_fullsized":"2024-04-23T21:21:52.955990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.300623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":35175237,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3119,"num_read_kb":22218,"num_write":136,"num_write_kb":6121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"47'141","reported_seq":3074,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983285+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:21:52.983285+0000","last_peered":"2024-04-23T21:21:52.983285+0000","last_clean":"2024-04-23T21:21:52.983285+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:21:52.983285+0000","last_undegraded":"2024-04-23T21:21:52.983285+0000","last_fullsized":"2024-04-23T21:21:52.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:11.541088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":20972537,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3351,"num_read_kb":11849,"num_write":140,"num_write_kb":2182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"47'205","reported_seq":1240,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982695+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:21:52.982695+0000","last_peered":"2024-04-23T21:21:52.982695+0000","last_clean":"2024-04-23T21:21:52.982695+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:21:52.982695+0000","last_undegraded":"2024-04-23T21:21:52.982695+0000","last_fullsized":"2024-04-23T21:21:52.982695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:01.195146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":34263604,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":1329,"num_read_kb":8056,"num_write":238,"num_write_kb":4087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"47'140","reported_seq":5899,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955883+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:21:52.955883+0000","last_peered":"2024-04-23T21:21:52.955883+0000","last_clean":"2024-04-23T21:21:52.955883+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:21:52.955883+0000","last_undegraded":"2024-04-23T21:21:52.955883+0000","last_fullsized":"2024-04-23T21:21:52.955883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:53:13.032035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":54386909,"num_objects":29,"num_object_clones":11,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":6,"num_read":5730,"num_read_kb":23530,"num_write":175,"num_write_kb":41084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"49'286","reported_seq":1974,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.051131+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:21:53.051131+0000","last_peered":"2024-04-23T21:21:53.051131+0000","last_clean":"2024-04-23T21:21:53.051131+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:21:53.051131+0000","last_undegraded":"2024-04-23T21:21:53.051131+0000","last_fullsized":"2024-04-23T21:21:53.051131+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:58:52.615083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":7192981,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3234,"num_read_kb":4291,"num_write":396,"num_write_kb":3443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"47'287","reported_seq":1171,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-04-23T21:21:51.824680+0000","last_change":"2024-04-23T21:21:52.936981+0000","last_active":"2024-04-23T21:21:52.936981+0000","last_peered":"2024-04-23T21:21:51.824680+0000","last_clean":"2024-04-23T21:21:16.757901+0000","last_became_active":"2024-04-23T21:21:50.952176+0000","last_became_peered":"2024-04-23T21:21:50.952176+0000","last_unstale":"2024-04-23T21:21:51.824680+0000","last_undegraded":"2024-04-23T21:21:51.824680+0000","last_fullsized":"2024-04-23T21:21:50.941229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21562016,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":5,"num_read":2283,"num_read_kb":8479,"num_write":421,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"49'365","reported_seq":2317,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955453+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:21:52.955453+0000","last_peered":"2024-04-23T21:21:52.955453+0000","last_clean":"2024-04-23T21:21:52.955453+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:21:52.955453+0000","last_undegraded":"2024-04-23T21:21:52.955453+0000","last_fullsized":"2024-04-23T21:21:52.955453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:22:06.591710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":18910114,"num_objects":22,"num_object_clones":6,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":7,"num_read":4168,"num_read_kb":10026,"num_write":478,"num_write_kb":1753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"47'144","reported_seq":1209,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983404+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:21:52.983404+0000","last_peered":"2024-04-23T21:21:52.983404+0000","last_clean":"2024-04-23T21:21:52.983404+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:21:52.983404+0000","last_undegraded":"2024-04-23T21:21:52.983404+0000","last_fullsized":"2024-04-23T21:21:52.983404+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:25:47.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":15859166,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":2060,"num_read_kb":3980,"num_write":149,"num_write_kb":4598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"47'83","reported_seq":749,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983438+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:21:52.983438+0000","last_peered":"2024-04-23T21:21:52.983438+0000","last_clean":"2024-04-23T21:21:52.983438+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:21:52.983438+0000","last_undegraded":"2024-04-23T21:21:52.983438+0000","last_fullsized":"2024-04-23T21:21:52.983438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.733855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":10650326,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":6,"num_read":589,"num_read_kb":6241,"num_write":84,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"47'179","reported_seq":3068,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982860+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:21:52.982860+0000","last_peered":"2024-04-23T21:21:52.982860+0000","last_clean":"2024-04-23T21:21:52.982860+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:21:52.982860+0000","last_undegraded":"2024-04-23T21:21:52.982860+0000","last_fullsized":"2024-04-23T21:21:52.982860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:55.813624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":38280150,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":8,"num_read":3417,"num_read_kb":10232,"num_write":174,"num_write_kb":3337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"49'165","reported_seq":1186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955660+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:21:52.955660+0000","last_peered":"2024-04-23T21:21:52.955660+0000","last_clean":"2024-04-23T21:21:52.955660+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:21:52.955660+0000","last_undegraded":"2024-04-23T21:21:52.955660+0000","last_fullsized":"2024-04-23T21:21:52.955660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:12.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":59076106,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1496,"num_read_kb":13901,"num_write":159,"num_write_kb":3684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153906+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:53.153906+0000","last_peered":"2024-04-23T21:21:53.153906+0000","last_clean":"2024-04-23T21:21:53.153906+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:53.153906+0000","last_undegraded":"2024-04-23T21:21:53.153906+0000","last_fullsized":"2024-04-23T21:21:53.153906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"47'185","reported_seq":735,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983321+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:21:52.983321+0000","last_peered":"2024-04-23T21:21:52.983321+0000","last_clean":"2024-04-23T21:21:52.983321+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:21:52.983321+0000","last_undegraded":"2024-04-23T21:21:52.983321+0000","last_fullsized":"2024-04-23T21:21:52.983321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:45.262475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":17199688,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":809,"num_read_kb":4718,"num_write":231,"num_write_kb":2251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153796+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:53.153796+0000","last_peered":"2024-04-23T21:21:53.153796+0000","last_clean":"2024-04-23T21:21:53.153796+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:53.153796+0000","last_undegraded":"2024-04-23T21:21:53.153796+0000","last_fullsized":"2024-04-23T21:21:53.153796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"47'132","reported_seq":968,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-04-23T21:21:51.585466+0000","last_change":"2024-04-23T21:21:52.937003+0000","last_active":"2024-04-23T21:21:52.937003+0000","last_peered":"2024-04-23T21:21:51.585466+0000","last_clean":"2024-04-23T21:21:16.757901+0000","last_became_active":"2024-04-23T21:21:50.944101+0000","last_became_peered":"2024-04-23T21:21:50.944101+0000","last_unstale":"2024-04-23T21:21:51.585466+0000","last_undegraded":"2024-04-23T21:21:51.585466+0000","last_fullsized":"2024-04-23T21:21:50.940234+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36817513,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":1011,"num_read_kb":4230,"num_write":129,"num_write_kb":1063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955727+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:52.955727+0000","last_peered":"2024-04-23T21:21:52.955727+0000","last_clean":"2024-04-23T21:21:52.955727+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:52.955727+0000","last_undegraded":"2024-04-23T21:21:52.955727+0000","last_fullsized":"2024-04-23T21:21:52.955727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"49'203","reported_seq":932,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.956233+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:21:52.956233+0000","last_peered":"2024-04-23T21:21:52.956233+0000","last_clean":"2024-04-23T21:21:52.956233+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:21:52.956233+0000","last_undegraded":"2024-04-23T21:21:52.956233+0000","last_fullsized":"2024-04-23T21:21:52.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:50.596282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":28518234,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":1212,"num_read_kb":5775,"num_write":304,"num_write_kb":2411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955645+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:52.955645+0000","last_peered":"2024-04-23T21:21:52.955645+0000","last_clean":"2024-04-23T21:21:52.955645+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:52.955645+0000","last_undegraded":"2024-04-23T21:21:52.955645+0000","last_fullsized":"2024-04-23T21:21:52.955645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"47'95","reported_seq":89,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954237+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:21:51.954237+0000","last_peered":"2024-04-23T21:21:51.954237+0000","last_clean":"2024-04-23T21:21:51.954237+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:21:51.954237+0000","last_undegraded":"2024-04-23T21:21:51.954237+0000","last_fullsized":"2024-04-23T21:21:51.954237+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:55.523850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":8301131,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":2115,"num_read_kb":4398,"num_write":98,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"47'101","reported_seq":582,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955437+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:21:52.955437+0000","last_peered":"2024-04-23T21:21:52.955437+0000","last_clean":"2024-04-23T21:21:52.955437+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:21:52.955437+0000","last_undegraded":"2024-04-23T21:21:52.955437+0000","last_fullsized":"2024-04-23T21:21:52.955437+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:58.720074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":58944694,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":414,"num_read_kb":6528,"num_write":108,"num_write_kb":2585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":233,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982664+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:21:52.982664+0000","last_peered":"2024-04-23T21:21:52.982664+0000","last_clean":"2024-04-23T21:21:52.982664+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:21:52.982664+0000","last_undegraded":"2024-04-23T21:21:52.982664+0000","last_fullsized":"2024-04-23T21:21:52.982664+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954872+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:21:51.954872+0000","last_peered":"2024-04-23T21:21:51.954872+0000","last_clean":"2024-04-23T21:21:51.954872+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:21:51.954872+0000","last_undegraded":"2024-04-23T21:21:51.954872+0000","last_fullsized":"2024-04-23T21:21:51.954872+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"47'65","reported_seq":333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954207+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:21:51.954207+0000","last_peered":"2024-04-23T21:21:51.954207+0000","last_clean":"2024-04-23T21:21:51.954207+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:21:51.954207+0000","last_undegraded":"2024-04-23T21:21:51.954207+0000","last_fullsized":"2024-04-23T21:21:51.954207+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:46:02.667761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":13458230,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2558,"num_read_kb":7343,"num_write":68,"num_write_kb":978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982867+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:52.982867+0000","last_peered":"2024-04-23T21:21:52.982867+0000","last_clean":"2024-04-23T21:21:52.982867+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:52.982867+0000","last_undegraded":"2024-04-23T21:21:52.982867+0000","last_fullsized":"2024-04-23T21:21:52.982867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"49'404","reported_seq":3420,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.054713+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:21:53.054713+0000","last_peered":"2024-04-23T21:21:53.054713+0000","last_clean":"2024-04-23T21:21:53.054713+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:21:53.054713+0000","last_undegraded":"2024-04-23T21:21:53.054713+0000","last_fullsized":"2024-04-23T21:21:53.054713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:05.091215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":39808083,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":4664,"num_read_kb":14579,"num_write":617,"num_write_kb":14556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982596+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:52.982596+0000","last_peered":"2024-04-23T21:21:52.982596+0000","last_clean":"2024-04-23T21:21:52.982596+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:52.982596+0000","last_undegraded":"2024-04-23T21:21:52.982596+0000","last_fullsized":"2024-04-23T21:21:52.982596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"47'320","reported_seq":2709,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955539+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:21:52.955539+0000","last_peered":"2024-04-23T21:21:52.955539+0000","last_clean":"2024-04-23T21:21:52.955539+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:21:52.955539+0000","last_undegraded":"2024-04-23T21:21:52.955539+0000","last_fullsized":"2024-04-23T21:21:52.955539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:38:46.948518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":29341447,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":3898,"num_read_kb":5955,"num_write":477,"num_write_kb":911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982773+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:52.982773+0000","last_peered":"2024-04-23T21:21:52.982773+0000","last_clean":"2024-04-23T21:21:52.982773+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:52.982773+0000","last_undegraded":"2024-04-23T21:21:52.982773+0000","last_fullsized":"2024-04-23T21:21:52.982773+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"47'86","reported_seq":774,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955803+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:21:52.955803+0000","last_peered":"2024-04-23T21:21:52.955803+0000","last_clean":"2024-04-23T21:21:52.955803+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:21:52.955803+0000","last_undegraded":"2024-04-23T21:21:52.955803+0000","last_fullsized":"2024-04-23T21:21:52.955803+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:02.722144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":10623714,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":807,"num_read_kb":5246,"num_write":99,"num_write_kb":4919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"49'124","reported_seq":3004,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955564+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:21:52.955564+0000","last_peered":"2024-04-23T21:21:52.955564+0000","last_clean":"2024-04-23T21:21:52.955564+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:21:52.955564+0000","last_undegraded":"2024-04-23T21:21:52.955564+0000","last_fullsized":"2024-04-23T21:21:52.955564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:47.925714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":36448087,"num_objects":21,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":2841,"num_read_kb":17558,"num_write":155,"num_write_kb":5925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"47'235","reported_seq":1482,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153699+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:21:53.153699+0000","last_peered":"2024-04-23T21:21:53.153699+0000","last_clean":"2024-04-23T21:21:53.153699+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:21:53.153699+0000","last_undegraded":"2024-04-23T21:21:53.153699+0000","last_fullsized":"2024-04-23T21:21:53.153699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:59.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":11922587,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":2760,"num_read_kb":6147,"num_write":247,"num_write_kb":4213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"49'128","reported_seq":996,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.968442+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:21:52.968442+0000","last_peered":"2024-04-23T21:21:52.968442+0000","last_clean":"2024-04-23T21:21:52.968442+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:21:52.968442+0000","last_undegraded":"2024-04-23T21:21:52.968442+0000","last_fullsized":"2024-04-23T21:21:52.968442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:43.869763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":25696550,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":1101,"num_read_kb":6618,"num_write":125,"num_write_kb":1046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"49'132","reported_seq":452,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982677+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:21:52.982677+0000","last_peered":"2024-04-23T21:21:52.982677+0000","last_clean":"2024-04-23T21:21:52.982677+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:21:52.982677+0000","last_undegraded":"2024-04-23T21:21:52.982677+0000","last_fullsized":"2024-04-23T21:21:52.982677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:57.761103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":33331654,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":333,"num_read_kb":1195,"num_write":138,"num_write_kb":1970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"47'97","reported_seq":842,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983003+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:21:52.983003+0000","last_peered":"2024-04-23T21:21:52.983003+0000","last_clean":"2024-04-23T21:21:52.983003+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:21:52.983003+0000","last_undegraded":"2024-04-23T21:21:52.983003+0000","last_fullsized":"2024-04-23T21:21:52.983003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:04.496297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":15715007,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":704,"num_read_kb":8502,"num_write":107,"num_write_kb":1863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"47'129","reported_seq":585,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982836+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:21:52.982836+0000","last_peered":"2024-04-23T21:21:52.982836+0000","last_clean":"2024-04-23T21:21:52.982836+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:21:52.982836+0000","last_undegraded":"2024-04-23T21:21:52.982836+0000","last_fullsized":"2024-04-23T21:21:52.982836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:20.765350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":13162577,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1033,"num_read_kb":1175,"num_write":131,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"47'615","reported_seq":4334,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982921+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:21:52.982921+0000","last_peered":"2024-04-23T21:21:52.982921+0000","last_clean":"2024-04-23T21:21:52.982921+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:21:52.982921+0000","last_undegraded":"2024-04-23T21:21:52.982921+0000","last_fullsized":"2024-04-23T21:21:52.982921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:54.368847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":41509021,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":6821,"num_read_kb":26226,"num_write":985,"num_write_kb":3821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"47'161","reported_seq":760,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153697+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:21:53.153697+0000","last_peered":"2024-04-23T21:21:53.153697+0000","last_clean":"2024-04-23T21:21:53.153697+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:21:53.153697+0000","last_undegraded":"2024-04-23T21:21:53.153697+0000","last_fullsized":"2024-04-23T21:21:53.153697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:24:25.016985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":54198751,"num_objects":15,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":783,"num_read_kb":8164,"num_write":203,"num_write_kb":4453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"47'82","reported_seq":484,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982701+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:21:52.982701+0000","last_peered":"2024-04-23T21:21:52.982701+0000","last_clean":"2024-04-23T21:21:52.982701+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:21:52.982701+0000","last_undegraded":"2024-04-23T21:21:52.982701+0000","last_fullsized":"2024-04-23T21:21:52.982701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:45.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":18303703,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":346,"num_read_kb":4600,"num_write":86,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"49'313","reported_seq":688,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983014+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:21:52.983014+0000","last_peered":"2024-04-23T21:21:52.983014+0000","last_clean":"2024-04-23T21:21:52.983014+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:21:52.983014+0000","last_undegraded":"2024-04-23T21:21:52.983014+0000","last_fullsized":"2024-04-23T21:21:52.983014+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:14.826652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":15101465,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":1568,"num_read_kb":2673,"num_write":458,"num_write_kb":4259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"47'192","reported_seq":559,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982856+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:21:52.982856+0000","last_peered":"2024-04-23T21:21:52.982856+0000","last_clean":"2024-04-23T21:21:52.982856+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:21:52.982856+0000","last_undegraded":"2024-04-23T21:21:52.982856+0000","last_fullsized":"2024-04-23T21:21:52.982856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:59.333608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":19208023,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":884,"num_read_kb":1857,"num_write":269,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":879735914,"num_objects":617,"num_object_clones":153,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":144,"num_read":70076,"num_read_kb":270885,"num_write":7859,"num_write_kb":142524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21934080,"data_stored":135041008,"data_compressed":7204616,"data_compressed_allocated":8769536,"data_compressed_original":123948610,"omap_allocated":0,"internal_metadata":0},"log_size":6138,"ondisk_log_size":6138,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39308,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332532,"statfs":{"total":96636764160,"available":96596512768,"internally_reserved":0,"allocated":4050944,"data_stored":15955423,"data_compressed":935477,"data_compressed_allocated":1191936,"data_compressed_original":13730463,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1482752,"data_stored":8767476,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38048,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333792,"statfs":{"total":96636764160,"available":96597803008,"internally_reserved":0,"allocated":2772992,"data_stored":11081187,"data_compressed":569971,"data_compressed_allocated":741376,"data_compressed_original":9640514,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44812,"kb_used_data":5312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94327028,"statfs":{"total":96636764160,"available":96590876672,"internally_reserved":0,"allocated":5439488,"data_stored":39778627,"data_compressed":2059767,"data_compressed_allocated":2392064,"data_compressed_original":37426513,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39260,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332580,"statfs":{"total":96636764160,"available":96596561920,"internally_reserved":0,"allocated":3989504,"data_stored":15856050,"data_compressed":843706,"data_compressed_allocated":1036288,"data_compressed_original":13573311,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":5846997,"data_compressed":295832,"data_compressed_allocated":380928,"data_compressed_original":4885611,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30808,"kb_used_data":2444,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341032,"statfs":{"total":96636764160,"available":96605216768,"internally_reserved":0,"allocated":2502656,"data_stored":12862045,"data_compressed":732887,"data_compressed_allocated":851968,"data_compressed_original":11698176,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39696,"kb_used_data":4352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332144,"statfs":{"total":96636764160,"available":96596115456,"internally_reserved":0,"allocated":4456448,"data_stored":28101784,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26089480,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3899392,"data_stored":27852352,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26088665,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1904640,"data_stored":12028102,"data_compressed":698829,"data_compressed_allocated":806912,"data_compressed_original":11108352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":5637883,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3411968,"data_stored":15537863,"data_compressed":840584,"data_compressed_allocated":1024000,"data_compressed_original":13507775,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4890624,"data_stored":39545650,"data_compressed":2060359,"data_compressed_allocated":2396160,"data_compressed_original":37438801,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2215936,"data_stored":10812057,"data_compressed":568802,"data_compressed_allocated":737280,"data_compressed_original":9615938,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":933888,"data_stored":8523901,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3440640,"data_stored":15103200,"data_compressed":902286,"data_compressed_allocated":1146880,"data_compressed_original":13139824,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:57.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:57.450 INFO:teuthology.orchestra.run.smithi111.stdout:2913 writesame 0xb36191e thru 0xb364263 data_size 0x9(0x2946 bytes) 2024-04-23T21:21:57.450 INFO:teuthology.orchestra.run.smithi111.stdout:2914 read 0x8dcce5e thru 0x8dd0055 (0x31f8 bytes) 2024-04-23T21:21:57.450 INFO:teuthology.orchestra.run.smithi111.stdout:2915 write 0x7d210a3 thru 0x7d29837 (0x8795 bytes) 2024-04-23T21:21:57.454 INFO:teuthology.orchestra.run.smithi111.stdout:2916 clone 24 order 18 su 131072 sc 10 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2140 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7372800~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.185032595000003, 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032102877000000002, 2024-04-23T21:21:57.467 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2140 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282175+0000", 2024-04-23T21:21:57.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999995e-07 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282177+0000", 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0379999999999998e-06 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282185+0000", 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9119999999999998e-06 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282190+0000", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7269999999999998e-06 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314126+0000", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.031936101000000001 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314204+0000", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7791000000000001e-05 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314278+0000", 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3696999999999996e-05 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.185006600999998, 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:21:57.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184986522999999, 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.477 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:57.481 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184965894000001, 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:57.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:57.485 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184940636, 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:57.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.489 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184914057, 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:21:57.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.494 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184893903000003, 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.495 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:21:57.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184868143000003, 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:21:57.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:21:57.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:21:57.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184836853999997, 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.504 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:21:57.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184822287000003, 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:21:57.508 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184802871000002, 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:21:57.512 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.516 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184787688, 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:21:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.520 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184777119000003, 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:21:57.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:21:57.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.525 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184767502, 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:21:57.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.18475797, 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.530 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:57.534 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.184744971999997, 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:21:57.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:21:57.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.538 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.057699014000001, 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:21:57.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:21:57.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.057680267999999, 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.546 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.057644277000001, 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:21:57.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:57.551 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 33.057616695, 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:57.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:21:57.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:21:57.556 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:21:57.556 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:57.557 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:21:57.558 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:57.557+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:57.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:21:57.658 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:21:57.659+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:21:57.686 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:57.687 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:57.701 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:21:57.701 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:21:57.701 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:21:57.701 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:21:57.703 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:21:57.704 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:21:57.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:21:57.720 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:57.721 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:21:57.737 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:21:57.737 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:21:57.742 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:57.743 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:57.753 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-04-23T21:21:57.601452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":880326282,"num_objects":619,"num_object_clones":153,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":144,"num_read":70192,"num_read_kb":270985,"num_write":8018,"num_write_kb":144940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6255,"ondisk_log_size":6255,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297488,"kb_used_data":25860,"kb_used_omap":17,"kb_used_meta":271470,"kb_avail":754677232,"statfs":{"total":773094113280,"available":772789485568,"internally_reserved":0,"allocated":26480640,"data_stored":138249589,"data_compressed":7273840,"data_compressed_allocated":8871936,"data_compressed_original":125170848,"omap_allocated":17904,"internal_metadata":277985808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63113314,"num_objects":87,"num_object_clones":33,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":28,"num_read":6368,"num_read_kb":57347,"num_write":645,"num_write_kb":15607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":4328802,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000703"},"pg_stats":[{"pgid":"3.1f","version":"47'202","reported_seq":2430,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983371+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:21:52.983371+0000","last_peered":"2024-04-23T21:21:52.983371+0000","last_clean":"2024-04-23T21:21:52.983371+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:21:52.983371+0000","last_undegraded":"2024-04-23T21:21:52.983371+0000","last_fullsized":"2024-04-23T21:21:52.983371+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:40:03.233990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":35796609,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":2638,"num_read_kb":4595,"num_write":274,"num_write_kb":2091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"47'142","reported_seq":3272,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955990+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:21:52.955990+0000","last_peered":"2024-04-23T21:21:52.955990+0000","last_clean":"2024-04-23T21:21:52.955990+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:21:52.955990+0000","last_undegraded":"2024-04-23T21:21:52.955990+0000","last_fullsized":"2024-04-23T21:21:52.955990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.300623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":35175237,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3119,"num_read_kb":22218,"num_write":136,"num_write_kb":6121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"47'141","reported_seq":3074,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983285+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:21:52.983285+0000","last_peered":"2024-04-23T21:21:52.983285+0000","last_clean":"2024-04-23T21:21:52.983285+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:21:52.983285+0000","last_undegraded":"2024-04-23T21:21:52.983285+0000","last_fullsized":"2024-04-23T21:21:52.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:11.541088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":20972537,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3351,"num_read_kb":11849,"num_write":140,"num_write_kb":2182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"47'205","reported_seq":1240,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982695+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:21:52.982695+0000","last_peered":"2024-04-23T21:21:52.982695+0000","last_clean":"2024-04-23T21:21:52.982695+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:21:52.982695+0000","last_undegraded":"2024-04-23T21:21:52.982695+0000","last_fullsized":"2024-04-23T21:21:52.982695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:01.195146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":34263604,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":1329,"num_read_kb":8056,"num_write":238,"num_write_kb":4087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"47'140","reported_seq":5899,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955883+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:21:52.955883+0000","last_peered":"2024-04-23T21:21:52.955883+0000","last_clean":"2024-04-23T21:21:52.955883+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:21:52.955883+0000","last_undegraded":"2024-04-23T21:21:52.955883+0000","last_fullsized":"2024-04-23T21:21:52.955883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:53:13.032035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":54386909,"num_objects":29,"num_object_clones":11,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":6,"num_read":5730,"num_read_kb":23530,"num_write":175,"num_write_kb":41084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"49'286","reported_seq":1974,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.051131+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:21:53.051131+0000","last_peered":"2024-04-23T21:21:53.051131+0000","last_clean":"2024-04-23T21:21:53.051131+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:21:53.051131+0000","last_undegraded":"2024-04-23T21:21:53.051131+0000","last_fullsized":"2024-04-23T21:21:53.051131+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:58:52.615083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":7192981,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3234,"num_read_kb":4291,"num_write":396,"num_write_kb":3443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"47'287","reported_seq":1171,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-04-23T21:21:51.824680+0000","last_change":"2024-04-23T21:21:52.936981+0000","last_active":"2024-04-23T21:21:52.936981+0000","last_peered":"2024-04-23T21:21:51.824680+0000","last_clean":"2024-04-23T21:21:16.757901+0000","last_became_active":"2024-04-23T21:21:50.952176+0000","last_became_peered":"2024-04-23T21:21:50.952176+0000","last_unstale":"2024-04-23T21:21:51.824680+0000","last_undegraded":"2024-04-23T21:21:51.824680+0000","last_fullsized":"2024-04-23T21:21:50.941229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21562016,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":5,"num_read":2283,"num_read_kb":8479,"num_write":421,"num_write_kb":2957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"49'365","reported_seq":2317,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955453+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:21:52.955453+0000","last_peered":"2024-04-23T21:21:52.955453+0000","last_clean":"2024-04-23T21:21:52.955453+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:21:52.955453+0000","last_undegraded":"2024-04-23T21:21:52.955453+0000","last_fullsized":"2024-04-23T21:21:52.955453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:22:06.591710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":18910114,"num_objects":22,"num_object_clones":6,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":7,"num_read":4168,"num_read_kb":10026,"num_write":478,"num_write_kb":1753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"47'144","reported_seq":1209,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983404+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:21:52.983404+0000","last_peered":"2024-04-23T21:21:52.983404+0000","last_clean":"2024-04-23T21:21:52.983404+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:21:52.983404+0000","last_undegraded":"2024-04-23T21:21:52.983404+0000","last_fullsized":"2024-04-23T21:21:52.983404+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:25:47.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":15859166,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":2060,"num_read_kb":3980,"num_write":149,"num_write_kb":4598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"47'83","reported_seq":749,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983438+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:21:52.983438+0000","last_peered":"2024-04-23T21:21:52.983438+0000","last_clean":"2024-04-23T21:21:52.983438+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:21:52.983438+0000","last_undegraded":"2024-04-23T21:21:52.983438+0000","last_fullsized":"2024-04-23T21:21:52.983438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.733855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":10650326,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":6,"num_read":589,"num_read_kb":6241,"num_write":84,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"47'179","reported_seq":3068,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982860+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:21:52.982860+0000","last_peered":"2024-04-23T21:21:52.982860+0000","last_clean":"2024-04-23T21:21:52.982860+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:21:52.982860+0000","last_undegraded":"2024-04-23T21:21:52.982860+0000","last_fullsized":"2024-04-23T21:21:52.982860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:55.813624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":38280150,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":8,"num_read":3417,"num_read_kb":10232,"num_write":174,"num_write_kb":3337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"49'165","reported_seq":1186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955660+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:21:52.955660+0000","last_peered":"2024-04-23T21:21:52.955660+0000","last_clean":"2024-04-23T21:21:52.955660+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:21:52.955660+0000","last_undegraded":"2024-04-23T21:21:52.955660+0000","last_fullsized":"2024-04-23T21:21:52.955660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:12.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":59076106,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1496,"num_read_kb":13901,"num_write":159,"num_write_kb":3684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153906+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:53.153906+0000","last_peered":"2024-04-23T21:21:53.153906+0000","last_clean":"2024-04-23T21:21:53.153906+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:53.153906+0000","last_undegraded":"2024-04-23T21:21:53.153906+0000","last_fullsized":"2024-04-23T21:21:53.153906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"47'185","reported_seq":735,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983321+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:21:52.983321+0000","last_peered":"2024-04-23T21:21:52.983321+0000","last_clean":"2024-04-23T21:21:52.983321+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:21:52.983321+0000","last_undegraded":"2024-04-23T21:21:52.983321+0000","last_fullsized":"2024-04-23T21:21:52.983321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:45.262475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":17199688,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":809,"num_read_kb":4718,"num_write":231,"num_write_kb":2251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153796+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:53.153796+0000","last_peered":"2024-04-23T21:21:53.153796+0000","last_clean":"2024-04-23T21:21:53.153796+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:53.153796+0000","last_undegraded":"2024-04-23T21:21:53.153796+0000","last_fullsized":"2024-04-23T21:21:53.153796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"47'132","reported_seq":968,"reported_epoch":48,"state":"creating+peering","last_fresh":"2024-04-23T21:21:51.585466+0000","last_change":"2024-04-23T21:21:52.937003+0000","last_active":"2024-04-23T21:21:52.937003+0000","last_peered":"2024-04-23T21:21:51.585466+0000","last_clean":"2024-04-23T21:21:16.757901+0000","last_became_active":"2024-04-23T21:21:50.944101+0000","last_became_peered":"2024-04-23T21:21:50.944101+0000","last_unstale":"2024-04-23T21:21:51.585466+0000","last_undegraded":"2024-04-23T21:21:51.585466+0000","last_fullsized":"2024-04-23T21:21:50.940234+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36817513,"num_objects":16,"num_object_clones":4,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":1011,"num_read_kb":4230,"num_write":129,"num_write_kb":1063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955727+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:52.955727+0000","last_peered":"2024-04-23T21:21:52.955727+0000","last_clean":"2024-04-23T21:21:52.955727+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:52.955727+0000","last_undegraded":"2024-04-23T21:21:52.955727+0000","last_fullsized":"2024-04-23T21:21:52.955727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"49'203","reported_seq":932,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.956233+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:21:52.956233+0000","last_peered":"2024-04-23T21:21:52.956233+0000","last_clean":"2024-04-23T21:21:52.956233+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:21:52.956233+0000","last_undegraded":"2024-04-23T21:21:52.956233+0000","last_fullsized":"2024-04-23T21:21:52.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:50.596282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":28518234,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":1212,"num_read_kb":5775,"num_write":304,"num_write_kb":2411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955645+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:52.955645+0000","last_peered":"2024-04-23T21:21:52.955645+0000","last_clean":"2024-04-23T21:21:52.955645+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:52.955645+0000","last_undegraded":"2024-04-23T21:21:52.955645+0000","last_fullsized":"2024-04-23T21:21:52.955645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"47'95","reported_seq":89,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954237+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:21:51.954237+0000","last_peered":"2024-04-23T21:21:51.954237+0000","last_clean":"2024-04-23T21:21:51.954237+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:21:51.954237+0000","last_undegraded":"2024-04-23T21:21:51.954237+0000","last_fullsized":"2024-04-23T21:21:51.954237+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:51:55.523850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":8301131,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":6,"num_read":2115,"num_read_kb":4398,"num_write":98,"num_write_kb":1245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"47'101","reported_seq":582,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955437+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:21:52.955437+0000","last_peered":"2024-04-23T21:21:52.955437+0000","last_clean":"2024-04-23T21:21:52.955437+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:21:52.955437+0000","last_undegraded":"2024-04-23T21:21:52.955437+0000","last_fullsized":"2024-04-23T21:21:52.955437+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:58.720074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":58944694,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":414,"num_read_kb":6528,"num_write":108,"num_write_kb":2585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":233,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982664+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:21:52.982664+0000","last_peered":"2024-04-23T21:21:52.982664+0000","last_clean":"2024-04-23T21:21:52.982664+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:21:52.982664+0000","last_undegraded":"2024-04-23T21:21:52.982664+0000","last_fullsized":"2024-04-23T21:21:52.982664+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":24,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954872+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:21:51.954872+0000","last_peered":"2024-04-23T21:21:51.954872+0000","last_clean":"2024-04-23T21:21:51.954872+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:21:51.954872+0000","last_undegraded":"2024-04-23T21:21:51.954872+0000","last_fullsized":"2024-04-23T21:21:51.954872+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"47'65","reported_seq":333,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-04-23T21:21:51.954207+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:21:51.954207+0000","last_peered":"2024-04-23T21:21:51.954207+0000","last_clean":"2024-04-23T21:21:51.954207+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:21:51.954207+0000","last_undegraded":"2024-04-23T21:21:51.954207+0000","last_fullsized":"2024-04-23T21:21:51.954207+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:46:02.667761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":13458230,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2558,"num_read_kb":7343,"num_write":68,"num_write_kb":978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982867+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:52.982867+0000","last_peered":"2024-04-23T21:21:52.982867+0000","last_clean":"2024-04-23T21:21:52.982867+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:52.982867+0000","last_undegraded":"2024-04-23T21:21:52.982867+0000","last_fullsized":"2024-04-23T21:21:52.982867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"49'404","reported_seq":3420,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.054713+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:21:53.054713+0000","last_peered":"2024-04-23T21:21:53.054713+0000","last_clean":"2024-04-23T21:21:53.054713+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:21:53.054713+0000","last_undegraded":"2024-04-23T21:21:53.054713+0000","last_fullsized":"2024-04-23T21:21:53.054713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:05.091215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":39808083,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":4664,"num_read_kb":14579,"num_write":617,"num_write_kb":14556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982596+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:52.982596+0000","last_peered":"2024-04-23T21:21:52.982596+0000","last_clean":"2024-04-23T21:21:52.982596+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:52.982596+0000","last_undegraded":"2024-04-23T21:21:52.982596+0000","last_fullsized":"2024-04-23T21:21:52.982596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"47'320","reported_seq":2709,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955539+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:21:52.955539+0000","last_peered":"2024-04-23T21:21:52.955539+0000","last_clean":"2024-04-23T21:21:52.955539+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:21:52.955539+0000","last_undegraded":"2024-04-23T21:21:52.955539+0000","last_fullsized":"2024-04-23T21:21:52.955539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:38:46.948518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":29341447,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":3898,"num_read_kb":5955,"num_write":477,"num_write_kb":911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982773+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:52.982773+0000","last_peered":"2024-04-23T21:21:52.982773+0000","last_clean":"2024-04-23T21:21:52.982773+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:52.982773+0000","last_undegraded":"2024-04-23T21:21:52.982773+0000","last_fullsized":"2024-04-23T21:21:52.982773+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"47'86","reported_seq":774,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955803+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:21:52.955803+0000","last_peered":"2024-04-23T21:21:52.955803+0000","last_clean":"2024-04-23T21:21:52.955803+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:21:52.955803+0000","last_undegraded":"2024-04-23T21:21:52.955803+0000","last_fullsized":"2024-04-23T21:21:52.955803+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:02.722144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":10623714,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":807,"num_read_kb":5246,"num_write":99,"num_write_kb":4919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"49'124","reported_seq":3004,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955564+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:21:52.955564+0000","last_peered":"2024-04-23T21:21:52.955564+0000","last_clean":"2024-04-23T21:21:52.955564+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:21:52.955564+0000","last_undegraded":"2024-04-23T21:21:52.955564+0000","last_fullsized":"2024-04-23T21:21:52.955564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:47.925714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":36448087,"num_objects":21,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":2841,"num_read_kb":17558,"num_write":155,"num_write_kb":5925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"47'235","reported_seq":1482,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153699+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:21:53.153699+0000","last_peered":"2024-04-23T21:21:53.153699+0000","last_clean":"2024-04-23T21:21:53.153699+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:21:53.153699+0000","last_undegraded":"2024-04-23T21:21:53.153699+0000","last_fullsized":"2024-04-23T21:21:53.153699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:59.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":11922587,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":2760,"num_read_kb":6147,"num_write":247,"num_write_kb":4213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"49'128","reported_seq":996,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.968442+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:21:52.968442+0000","last_peered":"2024-04-23T21:21:52.968442+0000","last_clean":"2024-04-23T21:21:52.968442+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:21:52.968442+0000","last_undegraded":"2024-04-23T21:21:52.968442+0000","last_fullsized":"2024-04-23T21:21:52.968442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:43.869763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":25696550,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":1101,"num_read_kb":6618,"num_write":125,"num_write_kb":1046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"49'132","reported_seq":452,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982677+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:21:52.982677+0000","last_peered":"2024-04-23T21:21:52.982677+0000","last_clean":"2024-04-23T21:21:52.982677+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:21:52.982677+0000","last_undegraded":"2024-04-23T21:21:52.982677+0000","last_fullsized":"2024-04-23T21:21:52.982677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:57.761103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":33331654,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":333,"num_read_kb":1195,"num_write":138,"num_write_kb":1970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"47'97","reported_seq":842,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983003+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:21:52.983003+0000","last_peered":"2024-04-23T21:21:52.983003+0000","last_clean":"2024-04-23T21:21:52.983003+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:21:52.983003+0000","last_undegraded":"2024-04-23T21:21:52.983003+0000","last_fullsized":"2024-04-23T21:21:52.983003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:04.496297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":15715007,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":704,"num_read_kb":8502,"num_write":107,"num_write_kb":1863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"47'129","reported_seq":585,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982836+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:21:52.982836+0000","last_peered":"2024-04-23T21:21:52.982836+0000","last_clean":"2024-04-23T21:21:52.982836+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:21:52.982836+0000","last_undegraded":"2024-04-23T21:21:52.982836+0000","last_fullsized":"2024-04-23T21:21:52.982836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:20.765350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":13162577,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1033,"num_read_kb":1175,"num_write":131,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"47'615","reported_seq":4334,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982921+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:21:52.982921+0000","last_peered":"2024-04-23T21:21:52.982921+0000","last_clean":"2024-04-23T21:21:52.982921+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:21:52.982921+0000","last_undegraded":"2024-04-23T21:21:52.982921+0000","last_fullsized":"2024-04-23T21:21:52.982921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:54.368847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":41509021,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":6821,"num_read_kb":26226,"num_write":985,"num_write_kb":3821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"47'161","reported_seq":760,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153697+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:21:53.153697+0000","last_peered":"2024-04-23T21:21:53.153697+0000","last_clean":"2024-04-23T21:21:53.153697+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:21:53.153697+0000","last_undegraded":"2024-04-23T21:21:53.153697+0000","last_fullsized":"2024-04-23T21:21:53.153697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:24:25.016985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":54198751,"num_objects":15,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":783,"num_read_kb":8164,"num_write":203,"num_write_kb":4453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"47'82","reported_seq":484,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982701+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:21:52.982701+0000","last_peered":"2024-04-23T21:21:52.982701+0000","last_clean":"2024-04-23T21:21:52.982701+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:21:52.982701+0000","last_undegraded":"2024-04-23T21:21:52.982701+0000","last_fullsized":"2024-04-23T21:21:52.982701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:45.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":18303703,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":346,"num_read_kb":4600,"num_write":86,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"49'313","reported_seq":688,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983014+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:21:52.983014+0000","last_peered":"2024-04-23T21:21:52.983014+0000","last_clean":"2024-04-23T21:21:52.983014+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:21:52.983014+0000","last_undegraded":"2024-04-23T21:21:52.983014+0000","last_fullsized":"2024-04-23T21:21:52.983014+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:14.826652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":15101465,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":1568,"num_read_kb":2673,"num_write":458,"num_write_kb":4259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"47'192","reported_seq":559,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982856+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:21:52.982856+0000","last_peered":"2024-04-23T21:21:52.982856+0000","last_clean":"2024-04-23T21:21:52.982856+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:21:52.982856+0000","last_undegraded":"2024-04-23T21:21:52.982856+0000","last_fullsized":"2024-04-23T21:21:52.982856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:59.333608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":19208023,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":884,"num_read_kb":1857,"num_write":269,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":879735914,"num_objects":617,"num_object_clones":153,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":144,"num_read":70076,"num_read_kb":270885,"num_write":7859,"num_write_kb":142524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21934080,"data_stored":135041008,"data_compressed":7204616,"data_compressed_allocated":8769536,"data_compressed_original":123948610,"omap_allocated":0,"internal_metadata":0},"log_size":6138,"ondisk_log_size":6138,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39308,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332532,"statfs":{"total":96636764160,"available":96596512768,"internally_reserved":0,"allocated":4050944,"data_stored":15955423,"data_compressed":935477,"data_compressed_allocated":1191936,"data_compressed_original":13730463,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1482752,"data_stored":8767476,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38048,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333792,"statfs":{"total":96636764160,"available":96597803008,"internally_reserved":0,"allocated":2772992,"data_stored":11081187,"data_compressed":569971,"data_compressed_allocated":741376,"data_compressed_original":9640514,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44812,"kb_used_data":5312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94327028,"statfs":{"total":96636764160,"available":96590876672,"internally_reserved":0,"allocated":5439488,"data_stored":39778627,"data_compressed":2059767,"data_compressed_allocated":2392064,"data_compressed_original":37426513,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39260,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332580,"statfs":{"total":96636764160,"available":96596561920,"internally_reserved":0,"allocated":3989504,"data_stored":15856050,"data_compressed":843706,"data_compressed_allocated":1036288,"data_compressed_original":13573311,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":5846997,"data_compressed":295832,"data_compressed_allocated":380928,"data_compressed_original":4885611,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30808,"kb_used_data":2444,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341032,"statfs":{"total":96636764160,"available":96605216768,"internally_reserved":0,"allocated":2502656,"data_stored":12862045,"data_compressed":732887,"data_compressed_allocated":851968,"data_compressed_original":11698176,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39696,"kb_used_data":4352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332144,"statfs":{"total":96636764160,"available":96596115456,"internally_reserved":0,"allocated":4456448,"data_stored":28101784,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26089480,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3899392,"data_stored":27852352,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26088665,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1904640,"data_stored":12028102,"data_compressed":698829,"data_compressed_allocated":806912,"data_compressed_original":11108352,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":5637883,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3411968,"data_stored":15537863,"data_compressed":840584,"data_compressed_allocated":1024000,"data_compressed_original":13507775,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4890624,"data_stored":39545650,"data_compressed":2060359,"data_compressed_allocated":2396160,"data_compressed_original":37438801,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2215936,"data_stored":10812057,"data_compressed":568802,"data_compressed_allocated":737280,"data_compressed_original":9615938,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":933888,"data_stored":8523901,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3440640,"data_stored":15103200,"data_compressed":902286,"data_compressed_allocated":1146880,"data_compressed_original":13139824,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:57.758 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-23T21:21:57.759 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:21:57.760 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:21:57.760 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:57.760+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:57.772 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:21:57.773 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:21:57.790 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:21:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:21:57.808 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:21:57.808 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:21:57.854 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:57.854 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:57.861 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:57.862+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:57.863 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:21:57.961 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:57.962+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:57.965 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:57.965 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:57.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:21:57.987 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:21:58.062 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:58.063+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:58.065 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone23 from 0xb364264 (overlap 0x43fa35b) to 0x0 2024-04-23T21:21:58.136 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.136 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.139 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:58.139 INFO:teuthology.orchestra.run.smithi111.stdout:{"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":60,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1713907308,"num_in_osds":8,"osd_in_since":1713907259,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":39},{"state_name":"creating+peering","count":2}],"num_pgs":41,"num_pools":3,"num_objects":619,"data_bytes":880326282,"bytes_used":304627712,"bytes_avail":772789485568,"bytes_total":773094113280,"inactive_pgs_ratio":0.048780485987663269,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":1082010,"recovering_keys_per_sec":8,"num_objects_recovered":63,"num_bytes_recovered":4328802,"num_keys_recovered":36,"read_bytes_sec":14678250,"write_bytes_sec":3994689,"read_op_per_sec":1591,"write_op_per_sec":161},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-23T21:20:57.392574+0000","services":{}},"progress_events":{}} 2024-04-23T21:21:58.147 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:21:58.156 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-04-23T21:21:58.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:21:58.158 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-04-23T21:21:58.159 INFO:teuthology.orchestra.run.smithi111.stdout:2918 punch from 0x401f7eb to 0x4022028, (0x283d bytes) 2024-04-23T21:21:58.163 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:58.164+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:58.165 INFO:teuthology.orchestra.run.smithi111.stdout:2919 trunc from 0xb364264 to 0x93384a1 2024-04-23T21:21:58.174 INFO:teuthology.orchestra.run.smithi111.stdout:2921 write 0x578bdc5 thru 0x5797f33 (0xc16f bytes) 2024-04-23T21:21:58.184 INFO:teuthology.orchestra.run.smithi111.stdout:2922 writesame 0xab7d4bb thru 0xab81cfa data_size 0x44(0x4840 bytes) 2024-04-23T21:21:58.187 INFO:teuthology.orchestra.run.smithi111.stdout:2923 trunc from 0xab81cfb to 0x7dceb22 2024-04-23T21:21:58.207 INFO:teuthology.orchestra.run.smithi111.stdout:2924 read 0x2f90835 thru 0x2f957ae (0x4f7a bytes) 2024-04-23T21:21:58.207 INFO:teuthology.orchestra.run.smithi111.stdout:2925 read 0x7b76b9d thru 0x7b7eb72 (0x7fd6 bytes) 2024-04-23T21:21:58.207 INFO:teuthology.orchestra.run.smithi111.stdout:2927 read 0x5d8ff6 thru 0x5dd458 (0x4463 bytes) 2024-04-23T21:21:58.208 INFO:teuthology.orchestra.run.smithi111.stdout:2928 write 0x458660e thru 0x45905ba (0x9fad bytes) 2024-04-23T21:21:58.210 INFO:teuthology.orchestra.run.smithi111.stdout:2930 read 0x276649c thru 0x2766a92 (0x5f7 bytes) 2024-04-23T21:21:58.212 INFO:teuthology.orchestra.run.smithi111.stdout:2931 trunc from 0x7dceb22 to 0x6efd565 2024-04-23T21:21:58.234 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:58.234 INFO:teuthology.orchestra.run.smithi111.stdout:2934 writesame 0x6963905 thru 0x6969944 data_size 0x70(0x6040 bytes) 2024-04-23T21:21:58.237 INFO:teuthology.orchestra.run.smithi111.stdout:2935 writesame 0x14b807d thru 0x14ba033 data_size 0x1(0x1fb7 bytes) 2024-04-23T21:21:58.245 INFO:teuthology.orchestra.run.smithi111.stdout:2936 trunc from 0x6efd565 to 0x57ad51a 2024-04-23T21:21:58.260 INFO:teuthology.orchestra.run.smithi111.stdout:2937 read 0x2a956e4 thru 0x2aa2c3f (0xd55c bytes) 2024-04-23T21:21:58.265 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:58.266+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:58.277 INFO:teuthology.orchestra.run.smithi111.stdout:2939 writesame 0xede71b6 thru 0xedf6b64 data_size 0x1(0xf9af bytes) 2024-04-23T21:21:58.277 INFO:teuthology.orchestra.run.smithi111.stdout:2942 read 0x708efdb thru 0x709e61e (0xf644 bytes) 2024-04-23T21:21:58.277 INFO:teuthology.orchestra.run.smithi111.stdout:2943 writesame 0x112a3cd thru 0x113798a data_size 0x2(0xd5be bytes) 2024-04-23T21:21:58.296 INFO:teuthology.orchestra.run.smithi111.stdout:2944 read 0x15fff08 thru 0x160dd51 (0xde4a bytes) 2024-04-23T21:21:58.297 INFO:teuthology.orchestra.run.smithi111.stdout:2946 write 0xd29eadb thru 0xd2a6f63 (0x8489 bytes) 2024-04-23T21:21:58.301 INFO:teuthology.orchestra.run.smithi111.stdout:2947 write 0xae543fa thru 0xae5ad90 (0x6997 bytes) 2024-04-23T21:21:58.301 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.301 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.304 INFO:teuthology.orchestra.run.smithi111.stdout:2948 clone 25 order 24 su 65536 sc 9 2024-04-23T21:21:58.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:21:58.365 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:58.366+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:58.414 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.414 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.423 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:21:58.467 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:58.468+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:58.470 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:21:58.471 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:21:58.483 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-04-23T21:21:57.987934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":893932700,"num_objects":626,"num_object_clones":155,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":146,"num_read":70312,"num_read_kb":273269,"num_write":8082,"num_write_kb":148217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6285,"ondisk_log_size":6285,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":298200,"kb_used_data":26568,"kb_used_omap":17,"kb_used_meta":271470,"kb_avail":754676520,"statfs":{"total":773094113280,"available":772788756480,"internally_reserved":0,"allocated":27205632,"data_stored":148196904,"data_compressed":7756967,"data_compressed_allocated":9322496,"data_compressed_original":134819508,"omap_allocated":17904,"internal_metadata":277985808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13606418,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":120,"num_read_kb":2284,"num_write":64,"num_write_kb":3277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025063"},"pg_stats":[{"pgid":"3.1f","version":"47'202","reported_seq":2430,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983371+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:21:52.983371+0000","last_peered":"2024-04-23T21:21:52.983371+0000","last_clean":"2024-04-23T21:21:52.983371+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:21:52.983371+0000","last_undegraded":"2024-04-23T21:21:52.983371+0000","last_fullsized":"2024-04-23T21:21:52.983371+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:40:03.233990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":35796609,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":2638,"num_read_kb":4595,"num_write":274,"num_write_kb":2091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"47'142","reported_seq":3272,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955990+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:21:52.955990+0000","last_peered":"2024-04-23T21:21:52.955990+0000","last_clean":"2024-04-23T21:21:52.955990+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:21:52.955990+0000","last_undegraded":"2024-04-23T21:21:52.955990+0000","last_fullsized":"2024-04-23T21:21:52.955990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:42.300623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":35175237,"num_objects":24,"num_object_clones":8,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":3119,"num_read_kb":22218,"num_write":136,"num_write_kb":6121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"47'141","reported_seq":3074,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983285+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:21:52.983285+0000","last_peered":"2024-04-23T21:21:52.983285+0000","last_clean":"2024-04-23T21:21:52.983285+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:21:52.983285+0000","last_undegraded":"2024-04-23T21:21:52.983285+0000","last_fullsized":"2024-04-23T21:21:52.983285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:47:11.541088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":20972537,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3351,"num_read_kb":11849,"num_write":140,"num_write_kb":2182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"47'205","reported_seq":1240,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982695+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:21:52.982695+0000","last_peered":"2024-04-23T21:21:52.982695+0000","last_clean":"2024-04-23T21:21:52.982695+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:21:52.982695+0000","last_undegraded":"2024-04-23T21:21:52.982695+0000","last_fullsized":"2024-04-23T21:21:52.982695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:01.195146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":34263604,"num_objects":24,"num_object_clones":5,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":5,"num_read":1329,"num_read_kb":8056,"num_write":238,"num_write_kb":4087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"47'140","reported_seq":5899,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955883+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:21:52.955883+0000","last_peered":"2024-04-23T21:21:52.955883+0000","last_clean":"2024-04-23T21:21:52.955883+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:21:52.955883+0000","last_undegraded":"2024-04-23T21:21:52.955883+0000","last_fullsized":"2024-04-23T21:21:52.955883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:53:13.032035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":54386909,"num_objects":29,"num_object_clones":11,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":6,"num_read":5730,"num_read_kb":23530,"num_write":175,"num_write_kb":41084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"49'286","reported_seq":1974,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.051131+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:21:53.051131+0000","last_peered":"2024-04-23T21:21:53.051131+0000","last_clean":"2024-04-23T21:21:53.051131+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:21:53.051131+0000","last_undegraded":"2024-04-23T21:21:53.051131+0000","last_fullsized":"2024-04-23T21:21:53.051131+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:58:52.615083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":7192981,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3234,"num_read_kb":4291,"num_write":396,"num_write_kb":3443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"53'305","reported_seq":1260,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.215357+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:21:57.215357+0000","last_peered":"2024-04-23T21:21:57.215357+0000","last_clean":"2024-04-23T21:21:57.215357+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:21:57.215357+0000","last_undegraded":"2024-04-23T21:21:57.215357+0000","last_fullsized":"2024-04-23T21:21:57.215357+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:13.768955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":28178742,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":2342,"num_read_kb":10646,"num_write":456,"num_write_kb":4763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"49'365","reported_seq":2317,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955453+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:21:52.955453+0000","last_peered":"2024-04-23T21:21:52.955453+0000","last_clean":"2024-04-23T21:21:52.955453+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:21:52.955453+0000","last_undegraded":"2024-04-23T21:21:52.955453+0000","last_fullsized":"2024-04-23T21:21:52.955453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:22:06.591710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":18910114,"num_objects":22,"num_object_clones":6,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":7,"num_read":4168,"num_read_kb":10026,"num_write":478,"num_write_kb":1753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"47'144","reported_seq":1209,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983404+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:21:52.983404+0000","last_peered":"2024-04-23T21:21:52.983404+0000","last_clean":"2024-04-23T21:21:52.983404+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:21:52.983404+0000","last_undegraded":"2024-04-23T21:21:52.983404+0000","last_fullsized":"2024-04-23T21:21:52.983404+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:25:47.348718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":15859166,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":2060,"num_read_kb":3980,"num_write":149,"num_write_kb":4598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"47'83","reported_seq":749,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983438+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:21:52.983438+0000","last_peered":"2024-04-23T21:21:52.983438+0000","last_clean":"2024-04-23T21:21:52.983438+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:21:52.983438+0000","last_undegraded":"2024-04-23T21:21:52.983438+0000","last_fullsized":"2024-04-23T21:21:52.983438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:08.733855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":10650326,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":6,"num_read":589,"num_read_kb":6241,"num_write":84,"num_write_kb":1408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"47'179","reported_seq":3068,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982860+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:21:52.982860+0000","last_peered":"2024-04-23T21:21:52.982860+0000","last_clean":"2024-04-23T21:21:52.982860+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:21:52.982860+0000","last_undegraded":"2024-04-23T21:21:52.982860+0000","last_fullsized":"2024-04-23T21:21:52.982860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:45:55.813624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":38280150,"num_objects":30,"num_object_clones":9,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":8,"num_read":3417,"num_read_kb":10232,"num_write":174,"num_write_kb":3337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"49'165","reported_seq":1186,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955660+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:21:52.955660+0000","last_peered":"2024-04-23T21:21:52.955660+0000","last_clean":"2024-04-23T21:21:52.955660+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:21:52.955660+0000","last_undegraded":"2024-04-23T21:21:52.955660+0000","last_fullsized":"2024-04-23T21:21:52.955660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:12.563000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":59076106,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1496,"num_read_kb":13901,"num_write":159,"num_write_kb":3684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153906+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:53.153906+0000","last_peered":"2024-04-23T21:21:53.153906+0000","last_clean":"2024-04-23T21:21:53.153906+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:53.153906+0000","last_undegraded":"2024-04-23T21:21:53.153906+0000","last_fullsized":"2024-04-23T21:21:53.153906+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"47'185","reported_seq":735,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983321+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:21:52.983321+0000","last_peered":"2024-04-23T21:21:52.983321+0000","last_clean":"2024-04-23T21:21:52.983321+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:21:52.983321+0000","last_undegraded":"2024-04-23T21:21:52.983321+0000","last_fullsized":"2024-04-23T21:21:52.983321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:29:45.262475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":17199688,"num_objects":19,"num_object_clones":3,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":809,"num_read_kb":4718,"num_write":231,"num_write_kb":2251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153796+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:53.153796+0000","last_peered":"2024-04-23T21:21:53.153796+0000","last_clean":"2024-04-23T21:21:53.153796+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:53.153796+0000","last_undegraded":"2024-04-23T21:21:53.153796+0000","last_fullsized":"2024-04-23T21:21:53.153796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"53'136","reported_seq":1012,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.351654+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:21:57.351654+0000","last_peered":"2024-04-23T21:21:57.351654+0000","last_clean":"2024-04-23T21:21:57.351654+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:21:57.351654+0000","last_undegraded":"2024-04-23T21:21:57.351654+0000","last_fullsized":"2024-04-23T21:21:57.351654+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:21:52.380644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":39963241,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":4,"num_read":1030,"num_read_kb":4288,"num_write":133,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955727+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:52.955727+0000","last_peered":"2024-04-23T21:21:52.955727+0000","last_clean":"2024-04-23T21:21:52.955727+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:52.955727+0000","last_undegraded":"2024-04-23T21:21:52.955727+0000","last_fullsized":"2024-04-23T21:21:52.955727+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"49'203","reported_seq":932,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.956233+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:21:52.956233+0000","last_peered":"2024-04-23T21:21:52.956233+0000","last_clean":"2024-04-23T21:21:52.956233+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:21:52.956233+0000","last_undegraded":"2024-04-23T21:21:52.956233+0000","last_fullsized":"2024-04-23T21:21:52.956233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:23:50.596282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":28518234,"num_objects":15,"num_object_clones":3,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":1212,"num_read_kb":5775,"num_write":304,"num_write_kb":2411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955645+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:52.955645+0000","last_peered":"2024-04-23T21:21:52.955645+0000","last_clean":"2024-04-23T21:21:52.955645+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:52.955645+0000","last_undegraded":"2024-04-23T21:21:52.955645+0000","last_fullsized":"2024-04-23T21:21:52.955645+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"53'98","reported_seq":139,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977305+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:21:56.977305+0000","last_peered":"2024-04-23T21:21:56.977305+0000","last_clean":"2024-04-23T21:21:56.977305+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:21:56.977305+0000","last_undegraded":"2024-04-23T21:21:56.977305+0000","last_fullsized":"2024-04-23T21:21:56.977305+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:41.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":12145095,"num_objects":18,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":6,"num_read":2150,"num_read_kb":4398,"num_write":113,"num_write_kb":2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"47'101","reported_seq":582,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955437+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:21:52.955437+0000","last_peered":"2024-04-23T21:21:52.955437+0000","last_clean":"2024-04-23T21:21:52.955437+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:21:52.955437+0000","last_undegraded":"2024-04-23T21:21:52.955437+0000","last_fullsized":"2024-04-23T21:21:52.955437+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:59:58.720074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":58944694,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":414,"num_read_kb":6528,"num_write":108,"num_write_kb":2585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":233,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982664+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:21:52.982664+0000","last_peered":"2024-04-23T21:21:52.982664+0000","last_clean":"2024-04-23T21:21:52.982664+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:21:52.982664+0000","last_undegraded":"2024-04-23T21:21:52.982664+0000","last_fullsized":"2024-04-23T21:21:52.982664+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":34,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"50'70","reported_seq":354,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:15.468280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":13458230,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2565,"num_read_kb":7402,"num_write":78,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982867+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:52.982867+0000","last_peered":"2024-04-23T21:21:52.982867+0000","last_clean":"2024-04-23T21:21:52.982867+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:52.982867+0000","last_undegraded":"2024-04-23T21:21:52.982867+0000","last_fullsized":"2024-04-23T21:21:52.982867+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"49'404","reported_seq":3420,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.054713+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:21:53.054713+0000","last_peered":"2024-04-23T21:21:53.054713+0000","last_clean":"2024-04-23T21:21:53.054713+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:21:53.054713+0000","last_undegraded":"2024-04-23T21:21:53.054713+0000","last_fullsized":"2024-04-23T21:21:53.054713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:32:05.091215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":39808083,"num_objects":28,"num_object_clones":8,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":6,"num_read":4664,"num_read_kb":14579,"num_write":617,"num_write_kb":14556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982596+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:52.982596+0000","last_peered":"2024-04-23T21:21:52.982596+0000","last_clean":"2024-04-23T21:21:52.982596+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:52.982596+0000","last_undegraded":"2024-04-23T21:21:52.982596+0000","last_fullsized":"2024-04-23T21:21:52.982596+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"47'320","reported_seq":2709,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955539+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:21:52.955539+0000","last_peered":"2024-04-23T21:21:52.955539+0000","last_clean":"2024-04-23T21:21:52.955539+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:21:52.955539+0000","last_undegraded":"2024-04-23T21:21:52.955539+0000","last_fullsized":"2024-04-23T21:21:52.955539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:38:46.948518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":29341447,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":3898,"num_read_kb":5955,"num_write":477,"num_write_kb":911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":78,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982773+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:52.982773+0000","last_peered":"2024-04-23T21:21:52.982773+0000","last_clean":"2024-04-23T21:21:52.982773+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:52.982773+0000","last_undegraded":"2024-04-23T21:21:52.982773+0000","last_fullsized":"2024-04-23T21:21:52.982773+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"47'86","reported_seq":774,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955803+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:21:52.955803+0000","last_peered":"2024-04-23T21:21:52.955803+0000","last_clean":"2024-04-23T21:21:52.955803+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:21:52.955803+0000","last_undegraded":"2024-04-23T21:21:52.955803+0000","last_fullsized":"2024-04-23T21:21:52.955803+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:29:02.722144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":10623714,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":4,"num_read":807,"num_read_kb":5246,"num_write":99,"num_write_kb":4919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"49'124","reported_seq":3004,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.955564+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:21:52.955564+0000","last_peered":"2024-04-23T21:21:52.955564+0000","last_clean":"2024-04-23T21:21:52.955564+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:21:52.955564+0000","last_undegraded":"2024-04-23T21:21:52.955564+0000","last_fullsized":"2024-04-23T21:21:52.955564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:46:47.925714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":36448087,"num_objects":21,"num_object_clones":8,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":2841,"num_read_kb":17558,"num_write":155,"num_write_kb":5925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"47'235","reported_seq":1482,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153699+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:21:53.153699+0000","last_peered":"2024-04-23T21:21:53.153699+0000","last_clean":"2024-04-23T21:21:53.153699+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:21:53.153699+0000","last_undegraded":"2024-04-23T21:21:53.153699+0000","last_fullsized":"2024-04-23T21:21:53.153699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:21:59.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":11922587,"num_objects":14,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":2760,"num_read_kb":6147,"num_write":247,"num_write_kb":4213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"49'128","reported_seq":996,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.968442+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:21:52.968442+0000","last_peered":"2024-04-23T21:21:52.968442+0000","last_clean":"2024-04-23T21:21:52.968442+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:21:52.968442+0000","last_undegraded":"2024-04-23T21:21:52.968442+0000","last_fullsized":"2024-04-23T21:21:52.968442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:43.869763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":25696550,"num_objects":21,"num_object_clones":5,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":1101,"num_read_kb":6618,"num_write":125,"num_write_kb":1046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"49'132","reported_seq":452,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982677+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:21:52.982677+0000","last_peered":"2024-04-23T21:21:52.982677+0000","last_clean":"2024-04-23T21:21:52.982677+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:21:52.982677+0000","last_undegraded":"2024-04-23T21:21:52.982677+0000","last_fullsized":"2024-04-23T21:21:52.982677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:05:57.761103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":33331654,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":6,"num_read":333,"num_read_kb":1195,"num_write":138,"num_write_kb":1970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"47'97","reported_seq":842,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983003+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:21:52.983003+0000","last_peered":"2024-04-23T21:21:52.983003+0000","last_clean":"2024-04-23T21:21:52.983003+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:21:52.983003+0000","last_undegraded":"2024-04-23T21:21:52.983003+0000","last_fullsized":"2024-04-23T21:21:52.983003+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:58:04.496297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":15715007,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":704,"num_read_kb":8502,"num_write":107,"num_write_kb":1863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"47'129","reported_seq":585,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982836+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:21:52.982836+0000","last_peered":"2024-04-23T21:21:52.982836+0000","last_clean":"2024-04-23T21:21:52.982836+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:21:52.982836+0000","last_undegraded":"2024-04-23T21:21:52.982836+0000","last_fullsized":"2024-04-23T21:21:52.982836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:20.765350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":13162577,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":1033,"num_read_kb":1175,"num_write":131,"num_write_kb":2624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"47'615","reported_seq":4334,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982921+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:21:52.982921+0000","last_peered":"2024-04-23T21:21:52.982921+0000","last_clean":"2024-04-23T21:21:52.982921+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:21:52.982921+0000","last_undegraded":"2024-04-23T21:21:52.982921+0000","last_fullsized":"2024-04-23T21:21:52.982921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:54.368847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":41509021,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":6821,"num_read_kb":26226,"num_write":985,"num_write_kb":3821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"47'161","reported_seq":760,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:53.153697+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:21:53.153697+0000","last_peered":"2024-04-23T21:21:53.153697+0000","last_clean":"2024-04-23T21:21:53.153697+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:21:53.153697+0000","last_undegraded":"2024-04-23T21:21:53.153697+0000","last_fullsized":"2024-04-23T21:21:53.153697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:24:25.016985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":54198751,"num_objects":15,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":783,"num_read_kb":8164,"num_write":203,"num_write_kb":4453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"47'82","reported_seq":484,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982701+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:21:52.982701+0000","last_peered":"2024-04-23T21:21:52.982701+0000","last_clean":"2024-04-23T21:21:52.982701+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:21:52.982701+0000","last_undegraded":"2024-04-23T21:21:52.982701+0000","last_fullsized":"2024-04-23T21:21:52.982701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:37:45.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":18303703,"num_objects":14,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":346,"num_read_kb":4600,"num_write":86,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"49'313","reported_seq":688,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.983014+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:21:52.983014+0000","last_peered":"2024-04-23T21:21:52.983014+0000","last_clean":"2024-04-23T21:21:52.983014+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:21:52.983014+0000","last_undegraded":"2024-04-23T21:21:52.983014+0000","last_fullsized":"2024-04-23T21:21:52.983014+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:14.826652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":15101465,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":1568,"num_read_kb":2673,"num_write":458,"num_write_kb":4259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"47'192","reported_seq":559,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-04-23T21:21:52.982856+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:21:52.982856+0000","last_peered":"2024-04-23T21:21:52.982856+0000","last_clean":"2024-04-23T21:21:52.982856+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:21:52.982856+0000","last_undegraded":"2024-04-23T21:21:52.982856+0000","last_fullsized":"2024-04-23T21:21:52.982856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:59.333608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":19208023,"num_objects":21,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":5,"num_read":884,"num_read_kb":1857,"num_write":269,"num_write_kb":1624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":893342332,"num_objects":624,"num_object_clones":155,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":146,"num_read":70196,"num_read_kb":273169,"num_write":7923,"num_write_kb":145801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22597632,"data_stored":144958088,"data_compressed":7687743,"data_compressed_allocated":9220096,"data_compressed_original":133597270,"omap_allocated":0,"internal_metadata":0},"log_size":6168,"ondisk_log_size":6168,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39308,"kb_used_data":3956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332532,"statfs":{"total":96636764160,"available":96596512768,"internally_reserved":0,"allocated":4050944,"data_stored":15955423,"data_compressed":935477,"data_compressed_allocated":1191936,"data_compressed_original":13730463,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1482752,"data_stored":8767476,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38048,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333792,"statfs":{"total":96636764160,"available":96597803008,"internally_reserved":0,"allocated":2772992,"data_stored":11081187,"data_compressed":569971,"data_compressed_allocated":741376,"data_compressed_original":9640514,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44812,"kb_used_data":5312,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94327028,"statfs":{"total":96636764160,"available":96590876672,"internally_reserved":0,"allocated":5439488,"data_stored":39778627,"data_compressed":2059767,"data_compressed_allocated":2392064,"data_compressed_original":37426513,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39260,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332580,"statfs":{"total":96636764160,"available":96596561920,"internally_reserved":0,"allocated":3989504,"data_stored":15856050,"data_compressed":843706,"data_compressed_allocated":1036288,"data_compressed_original":13573311,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,32],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1785856,"data_stored":5846997,"data_compressed":295832,"data_compressed_allocated":380928,"data_compressed_original":4885611,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":3152,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":3227648,"data_stored":22809360,"data_compressed":1216014,"data_compressed_allocated":1302528,"data_compressed_original":21346836,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39696,"kb_used_data":4352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332144,"statfs":{"total":96636764160,"available":96596115456,"internally_reserved":0,"allocated":4456448,"data_stored":28101784,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26089480,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3899392,"data_stored":27852352,"data_compressed":1443674,"data_compressed_allocated":1847296,"data_compressed_original":26088665,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2568192,"data_stored":21945182,"data_compressed":1181956,"data_compressed_allocated":1257472,"data_compressed_original":20757012,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1236992,"data_stored":5637883,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3411968,"data_stored":15537863,"data_compressed":840584,"data_compressed_allocated":1024000,"data_compressed_original":13507775,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4890624,"data_stored":39545650,"data_compressed":2060359,"data_compressed_allocated":2396160,"data_compressed_original":37438801,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2215936,"data_stored":10812057,"data_compressed":568802,"data_compressed_allocated":737280,"data_compressed_original":9615938,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":933888,"data_stored":8523901,"data_compressed":392526,"data_compressed_allocated":430080,"data_compressed_original":8126780,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3440640,"data_stored":15103200,"data_compressed":902286,"data_compressed_allocated":1146880,"data_compressed_original":13139824,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:21:58.528 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.528 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:21:58.580 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:58.581+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:58.638 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.639 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:21:58.689 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:58.690+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:58.769 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:58.770+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:58.791 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.791 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:21:58.885 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:58.886+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:58.943 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:58.943 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:58.952 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:21:58.988 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:21:58.988 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:21:58.989+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:21:59.073 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:59.074+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:59.073 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone24 from 0xedf6b65 (overlap 0x57ad51a) to 0x1a4d987 2024-04-23T21:21:59.105 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.105 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.117 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:21:59.201 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:59.202+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:59.275 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:59.276+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:59.294 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.294 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.303 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:21:59.317 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-04-23T21:21:59.319 INFO:teuthology.orchestra.run.smithi111.stdout:2949 punch from 0xdf052a0 to 0xdf0f156, (0x9eb6 bytes) 2024-04-23T21:21:59.328 INFO:teuthology.orchestra.run.smithi111.stdout:2950 trunc from 0xedf6b65 to 0x9021eba 2024-04-23T21:21:59.339 INFO:teuthology.orchestra.run.smithi111.stdout:2951 read 0x7e061cc thru 0x7e10e9a (0xaccf bytes) 2024-04-23T21:21:59.339 INFO:teuthology.orchestra.run.smithi111.stdout:2953 write 0x1e6ecae thru 0x1e73e72 (0x51c5 bytes) 2024-04-23T21:21:59.376 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:21:59.377+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:21:59.393 INFO:teuthology.orchestra.run.smithi111.stdout:2954 punch from 0x8334334 to 0x834341b, (0xf0e7 bytes) 2024-04-23T21:21:59.393 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.393 INFO:teuthology.orchestra.run.smithi111.stdout:2956 read 0x4f92b2c thru 0x4f93d7c (0x1251 bytes) 2024-04-23T21:21:59.393 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.400 INFO:teuthology.orchestra.run.smithi111.stdout:2959 write 0xa59c41c thru 0xa5a8c52 (0xc837 bytes) 2024-04-23T21:21:59.414 INFO:teuthology.orchestra.run.smithi111.stdout:2960 writesame 0xc43a9db thru 0xc447d2f data_size 0x1(0xd355 bytes) 2024-04-23T21:21:59.414 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.414 INFO:teuthology.orchestra.run.smithi111.stdout:2963 write 0xbcfeda6 thru 0xbd0630d (0x7568 bytes) 2024-04-23T21:21:59.421 INFO:teuthology.orchestra.run.smithi111.stdout:2964 punch from 0xb03db39 to 0xb040a97, (0x2f5e bytes) 2024-04-23T21:21:59.422 INFO:teuthology.orchestra.run.smithi111.stdout:2966 write 0x54bc796 thru 0x54c9fc1 (0xd82c bytes) 2024-04-23T21:21:59.444 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.444 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.455 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:21:59.466 INFO:teuthology.orchestra.run.smithi111.stdout:2968 write 0xad51ada thru 0xad5a92b (0x8e52 bytes) 2024-04-23T21:21:59.468 INFO:teuthology.orchestra.run.smithi111.stdout:2969 write 0xb5d7735 thru 0xb5df0e2 (0x79ae bytes) 2024-04-23T21:21:59.480 INFO:teuthology.orchestra.run.smithi111.stdout:2971 trunc from 0xc447d30 to 0x2ef66e5 2024-04-23T21:21:59.481 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:21:59.482+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:21:59.561 INFO:teuthology.orchestra.run.smithi111.stdout:2974 punch from 0x2e9d580 to 0x2ea683e, (0x92be bytes) 2024-04-23T21:21:59.561 INFO:teuthology.orchestra.run.smithi111.stdout:2976 punch from 0x18a2a43 to 0x18a8948, (0x5f05 bytes) 2024-04-23T21:21:59.561 INFO:teuthology.orchestra.run.smithi111.stdout:2977 punch from 0x1e5894 to 0x1f297e, (0xd0ea bytes) 2024-04-23T21:21:59.561 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.561 INFO:teuthology.orchestra.run.smithi111.stdout:2980 read 0xef38ca thru 0xf00d8f (0xd4c6 bytes) 2024-04-23T21:21:59.562 INFO:teuthology.orchestra.run.smithi111.stdout:2981 punch from 0x838cad to 0x83aa9b, (0x1dee bytes) 2024-04-23T21:21:59.562 INFO:teuthology.orchestra.run.smithi111.stdout:2983 punch from 0x1d663a1 to 0x1d6eb57, (0x87b6 bytes) 2024-04-23T21:21:59.562 INFO:teuthology.orchestra.run.smithi111.stdout:2984 write 0x2b416f7 thru 0x2b448a5 (0x31af bytes) 2024-04-23T21:21:59.570 INFO:teuthology.orchestra.run.smithi111.stdout:2985 trunc from 0x2ef66e5 to 0x8da5658 2024-04-23T21:21:59.575 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.575 INFO:teuthology.orchestra.run.smithi111.stdout:2988 read 0x3eb79d0 thru 0x3ec6b87 (0xf1b8 bytes) 2024-04-23T21:21:59.577 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:21:59.578+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:21:59.583 INFO:teuthology.orchestra.run.smithi111.stdout:2989 writesame 0xaf55986 thru 0xaf5ba30 data_size 0x49(0x60ab bytes) 2024-04-23T21:21:59.591 INFO:teuthology.orchestra.run.smithi111.stdout:2990 trunc from 0xaf5ba31 to 0x40eb84d 2024-04-23T21:21:59.599 INFO:teuthology.orchestra.run.smithi111.stdout:2991 read 0xdc0b04 thru 0xdca76a (0x9c67 bytes) 2024-04-23T21:21:59.599 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.599 INFO:teuthology.orchestra.run.smithi111.stdout:2994 punch from 0x1b0770c to 0x1b0b036, (0x392a bytes) 2024-04-23T21:21:59.601 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.601 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.614 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:21:59.616 INFO:teuthology.orchestra.run.smithi111.stdout:2995 write 0xce4d563 thru 0xce58fea (0xba88 bytes) 2024-04-23T21:21:59.623 INFO:teuthology.orchestra.run.smithi111.stdout:2996 punch from 0x6ed8791 to 0x6ee4369, (0xbbd8 bytes) 2024-04-23T21:21:59.623 INFO:teuthology.orchestra.run.smithi111.stdout:2997 punch from 0x1668e8 to 0x16998a, (0x30a2 bytes) 2024-04-23T21:21:59.623 INFO:teuthology.orchestra.run.smithi111.stdout:2999 write 0xa167ec8 thru 0xa16d671 (0x57aa bytes) 2024-04-23T21:21:59.626 INFO:teuthology.orchestra.run.smithi111.stdout:3000 read 0x8960e03 thru 0x8964730 (0x392e bytes) 2024-04-23T21:21:59.626 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.627 INFO:teuthology.orchestra.run.smithi111.stdout:3003 write 0x9cb7c11 thru 0x9cbd1f1 (0x55e1 bytes) 2024-04-23T21:21:59.630 INFO:teuthology.orchestra.run.smithi111.stdout:3005 read 0x1154cb9 thru 0x11606bf (0xba07 bytes) 2024-04-23T21:21:59.630 INFO:teuthology.orchestra.run.smithi111.stdout:3007 write 0x291da0f thru 0x2924897 (0x6e89 bytes) 2024-04-23T21:21:59.632 INFO:teuthology.orchestra.run.smithi111.stdout:3008 write 0x1bf157 thru 0x1c1b5d (0x2a07 bytes) 2024-04-23T21:21:59.633 INFO:teuthology.orchestra.run.smithi111.stdout:3009 punch from 0x733d5d6 to 0x734842d, (0xae57 bytes) 2024-04-23T21:21:59.633 INFO:teuthology.orchestra.run.smithi111.stdout:3010 read 0x2581b48 thru 0x2590b05 (0xefbe bytes) 2024-04-23T21:21:59.634 INFO:teuthology.orchestra.run.smithi111.stdout:3011 write 0xbf8a909 thru 0xbf9124f (0x6947 bytes) 2024-04-23T21:21:59.637 INFO:teuthology.orchestra.run.smithi111.stdout:3012 read 0x6441630 thru 0x6450b4a (0xf51b bytes) 2024-04-23T21:21:59.646 INFO:teuthology.orchestra.run.smithi111.stdout:3013 write 0xed3f747 thru 0xed48d98 (0x9652 bytes) 2024-04-23T21:21:59.647 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.647 INFO:teuthology.orchestra.run.smithi111.stdout:3015 write 0x48f695 thru 0x4928e3 (0x324f bytes) 2024-04-23T21:21:59.649 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.649 INFO:teuthology.orchestra.run.smithi111.stdout:3017 writesame 0xda605f5 thru 0xda6641c data_size 0x5c(0x5e28 bytes) 2024-04-23T21:21:59.651 INFO:teuthology.orchestra.run.smithi111.stdout:3018 write 0x5f62b2f thru 0x5f7279c (0xfc6e bytes) 2024-04-23T21:21:59.653 INFO:teuthology.orchestra.run.smithi111.stdout:3019 read 0x6a95c3a thru 0x6aa2fb5 (0xd37c bytes) 2024-04-23T21:21:59.653 INFO:teuthology.orchestra.run.smithi111.stdout:3020 read 0x6519b45 thru 0x6521aba (0x7f76 bytes) 2024-04-23T21:21:59.654 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.654 INFO:teuthology.orchestra.run.smithi111.stdout:3022 write 0x83af33f thru 0x83b93c0 (0xa082 bytes) 2024-04-23T21:21:59.655 INFO:teuthology.orchestra.run.smithi111.stdout:3023 write 0x14d9b53 thru 0x14dbdd6 (0x2284 bytes) 2024-04-23T21:21:59.657 INFO:teuthology.orchestra.run.smithi111.stdout:3024 read 0xb42e8d2 thru 0xb43c007 (0xd736 bytes) 2024-04-23T21:21:59.667 INFO:teuthology.orchestra.run.smithi111.stdout:3025 trunc from 0xed48d99 to 0x630ca01 2024-04-23T21:21:59.678 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:21:59.680+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:21:59.679 INFO:teuthology.orchestra.run.smithi111.stdout:3028 read 0xac6679 thru 0xaccf15 (0x689d bytes) 2024-04-23T21:21:59.696 INFO:teuthology.orchestra.run.smithi111.stdout:3030 write 0xebba4a2 thru 0xebc150c (0x706b bytes) 2024-04-23T21:21:59.700 INFO:teuthology.orchestra.run.smithi111.stdout:3031 write 0xa49de64 thru 0xa49e1bf (0x35c bytes) 2024-04-23T21:21:59.704 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.704 INFO:teuthology.orchestra.run.smithi111.stdout:3033 write 0x2937af3 thru 0x2946ece (0xf3dc bytes) 2024-04-23T21:21:59.705 INFO:teuthology.orchestra.run.smithi111.stdout:3034 read 0x26c8077 thru 0x26cecef (0x6c79 bytes) 2024-04-23T21:21:59.706 INFO:teuthology.orchestra.run.smithi111.stdout:3035 write 0xc610d80 thru 0xc61a256 (0x94d7 bytes) 2024-04-23T21:21:59.707 INFO:teuthology.orchestra.run.smithi111.stdout:3036 punch from 0x2712c23 to 0x27199ec, (0x6dc9 bytes) 2024-04-23T21:21:59.707 INFO:teuthology.orchestra.run.smithi111.stdout:3037 write 0x312a48e thru 0x31397d0 (0xf343 bytes) 2024-04-23T21:21:59.709 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.709 INFO:teuthology.orchestra.run.smithi111.stdout:3039 punch from 0xc4364b7 to 0xc43a722, (0x426b bytes) 2024-04-23T21:21:59.721 INFO:teuthology.orchestra.run.smithi111.stdout:3041 trunc from 0xebc150d to 0x4b4eca8 2024-04-23T21:21:59.733 INFO:teuthology.orchestra.run.smithi111.stdout:3042 trunc from 0x4b4eca8 to 0x4fb4ce0 2024-04-23T21:21:59.734 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.734 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.738 INFO:teuthology.orchestra.run.smithi111.stdout:3043 read 0xe681a0 thru 0xe68b86 (0x9e7 bytes) 2024-04-23T21:21:59.739 INFO:teuthology.orchestra.run.smithi111.stdout:3044 read 0xc60419 thru 0xc679de (0x75c6 bytes) 2024-04-23T21:21:59.745 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:21:59.746 INFO:teuthology.orchestra.run.smithi111.stdout:3045 trunc from 0x4fb4ce0 to 0xa4bc6b9 2024-04-23T21:21:59.750 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.750 INFO:teuthology.orchestra.run.smithi111.stdout:3047 read 0x7081896 thru 0x7087a30 (0x619b bytes) 2024-04-23T21:21:59.751 INFO:teuthology.orchestra.run.smithi111.stdout:3048 read 0x585c4 thru 0x602e0 (0x7d1d bytes) 2024-04-23T21:21:59.751 INFO:teuthology.orchestra.run.smithi111.stdout:3049 write 0xc113e7 thru 0xc1c778 (0xb392 bytes) 2024-04-23T21:21:59.753 INFO:teuthology.orchestra.run.smithi111.stdout:3050 write 0x1bec4a2 thru 0x1bf6936 (0xa495 bytes) 2024-04-23T21:21:59.754 INFO:teuthology.orchestra.run.smithi111.stdout:3051 write 0x602da77 thru 0x60376bd (0x9c47 bytes) 2024-04-23T21:21:59.756 INFO:teuthology.orchestra.run.smithi111.stdout:3052 write 0x9298515 thru 0x929b806 (0x32f2 bytes) 2024-04-23T21:21:59.759 INFO:teuthology.orchestra.run.smithi111.stdout:3053 read 0x15b594b thru 0x15bfad4 (0xa18a bytes) 2024-04-23T21:21:59.766 INFO:teuthology.orchestra.run.smithi111.stdout:3054 write 0xbcfe553 thru 0xbcffb8a (0x1638 bytes) 2024-04-23T21:21:59.769 INFO:teuthology.orchestra.run.smithi111.stdout:3059 writesame 0x2784cdb thru 0x278d78c data_size 0x29(0x8ab2 bytes) 2024-04-23T21:21:59.770 INFO:teuthology.orchestra.run.smithi111.stdout:3060 write 0x7ca0a39 thru 0x7ca8aa6 (0x806e bytes) 2024-04-23T21:21:59.772 INFO:teuthology.orchestra.run.smithi111.stdout:3061 write 0x73f3955 thru 0x73f4423 (0xacf bytes) 2024-04-23T21:21:59.773 INFO:teuthology.orchestra.run.smithi111.stdout:3062 write 0x456e5cc thru 0x457d463 (0xee98 bytes) 2024-04-23T21:21:59.775 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.775 INFO:teuthology.orchestra.run.smithi111.stdout:3064 punch from 0x1049ab9 to 0x1057339, (0xd880 bytes) 2024-04-23T21:21:59.775 INFO:teuthology.orchestra.run.smithi111.stdout:3066 write 0x2d66eb9 thru 0x2d6df94 (0x70dc bytes) 2024-04-23T21:21:59.779 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:59.781+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:59.780 INFO:teuthology.orchestra.run.smithi111.stdout:3067 trunc from 0xbcffb8b to 0xed19340 2024-04-23T21:21:59.786 INFO:teuthology.orchestra.run.smithi111.stdout:3068 trunc from 0xed19340 to 0xd92e11f 2024-04-23T21:21:59.794 INFO:teuthology.orchestra.run.smithi111.stdout:3069 write 0x1886009 thru 0x188d673 (0x766b bytes) 2024-04-23T21:21:59.795 INFO:teuthology.orchestra.run.smithi111.stdout:3071 writesame 0x8886883 thru 0x888781b data_size 0xb(0xf99 bytes) 2024-04-23T21:21:59.796 INFO:teuthology.orchestra.run.smithi111.stdout:3072 punch from 0xbd6fa28 to 0xbd775f5, (0x7bcd bytes) 2024-04-23T21:21:59.796 INFO:teuthology.orchestra.run.smithi111.stdout:3073 write 0x808259f thru 0x808ab96 (0x85f8 bytes) 2024-04-23T21:21:59.801 INFO:teuthology.orchestra.run.smithi111.stdout:3074 trunc from 0xd92e11f to 0xac2ce91 2024-04-23T21:21:59.811 INFO:teuthology.orchestra.run.smithi111.stdout:3075 trunc from 0xac2ce91 to 0x78ba36e 2024-04-23T21:21:59.820 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:21:59.829 INFO:teuthology.orchestra.run.smithi111.stdout:3076 trunc from 0x78ba36e to 0xbfb6d0c 2024-04-23T21:21:59.835 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:21:59.836 INFO:teuthology.orchestra.run.smithi111.stdout:3079 writesame 0x878dfc9 thru 0x8793ebb data_size 0x1(0x5ef3 bytes) 2024-04-23T21:21:59.843 INFO:teuthology.orchestra.run.smithi111.stdout:3080 clone 26 order 24 su 8388608 sc 15 2024-04-23T21:21:59.864 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:21:59.864 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:21:59.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:21:59.915 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:21:59.916+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:21:59.943 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:21:59.943 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:21:59.977 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:21:59.977 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:21:59.981 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:21:59.981+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:21:59.996 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:21:59.996 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:00.029 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:00.030 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:00.045 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:00.045 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:00.045 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:00.045 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:00.045 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:00.047 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:00.047 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:00.056 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:00.068 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:00.068 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:00.081 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:00.082+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:00.084 INFO:teuthology.orchestra.run.smithi111.stdout:flattening image image_client.0-clone25 2024-04-23T21:22:00.086 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:00.086 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:00.105 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:00.105 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:00.179 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:00.179 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:00.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.538823344000001, 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.180 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.181 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.185 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.538791237, 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.538767850999999, 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.190 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.538747544, 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:00.195 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:00.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:00.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:00.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:00.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.204 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.538727732, 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.205 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.206 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:00.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.206 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:00.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537656333999999, 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:00.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537622989000001, 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.222 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.53759146, 2024-04-23T21:22:00.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.224 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:00.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:00.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:00.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:00.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.231 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537564071, 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:00.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:00.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537552164999999, 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:00.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:00.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:00.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:00.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.241 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537533921, 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.242 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:00.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537524103999999, 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.246 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:00.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:00.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:00.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537513857, 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:00.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:00.255 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537504205999999, 2024-04-23T21:22:00.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:00.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:00.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:00.263 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.537470707000001, 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:00.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:00.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.270 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.533845283, 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.533827413999999, 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.275 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.533808685, 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.279 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:00.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:00.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.533790257, 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.533772352, 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:00.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:00.292 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:00.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:00.294 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-04-23T21:22:00.294 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:00.294+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:00.304 INFO:teuthology.orchestra.run.smithi111.stdout:3081 read 0x35acf4b thru 0x35b897d (0xba33 bytes) 2024-04-23T21:22:00.306 INFO:teuthology.orchestra.run.smithi111.stdout:3082 writesame 0x57e4662 thru 0x57eb0d4 data_size 0x11(0x6a73 bytes) 2024-04-23T21:22:00.315 INFO:teuthology.orchestra.run.smithi111.stdout:3084 punch from 0x859abac to 0x859f8dd, (0x4d31 bytes) 2024-04-23T21:22:00.315 INFO:teuthology.orchestra.run.smithi111.stdout:3085 punch from 0x10f2cd to 0x1101fd, (0xf30 bytes) 2024-04-23T21:22:00.315 INFO:teuthology.orchestra.run.smithi111.stdout:3087 write 0x90c5bdd thru 0x90cee9a (0x92be bytes) 2024-04-23T21:22:00.340 INFO:teuthology.orchestra.run.smithi111.stdout:3088 read 0x4e9fccf thru 0x4ea0673 (0x9a5 bytes) 2024-04-23T21:22:00.341 INFO:teuthology.orchestra.run.smithi111.stdout:3089 trunc from 0xbfb6d0c to 0xc7b575b 2024-04-23T21:22:00.359 INFO:teuthology.orchestra.run.smithi111.stdout:3090 trunc from 0xc7b575b to 0x2df31f6 2024-04-23T21:22:00.394 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:00.395+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:00.421 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:00.421 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:00.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.421 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:61148 3.13 e56/18 3:cc4c0078:::rbd_data.1118e08f6c14.0000000000000002:head v 56'212, mlcod=55'210)", 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.408920+0000", 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012853435999999999, 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012897713, 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408920+0000", 2024-04-23T21:22:00.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408920+0000", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408922+0000", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.024e-06 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.416628+0000", 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.007705361 2024-04-23T21:22:00.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.416673+0000", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5265000000000001e-05 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.416695+0000", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2526000000000001e-05 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.416865+0000", 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00017014999999999999 2024-04-23T21:22:00.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.416933+0000", 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7077999999999997e-05 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:61150 3.a e56/18 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head v 56'317, mlcod=56'315)", 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.414706+0000", 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0070671839999999998, 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071827310000000004, 2024-04-23T21:22:00.425 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.414706+0000", 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.414706+0000", 2024-04-23T21:22:00.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.414709+0000", 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5969999999999999e-06 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.420265+0000", 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055558639999999998 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.420288+0000", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2736000000000001e-05 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.420302+0000", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4059e-05 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.420405+0000", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010331 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.420433+0000", 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8014e-05 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-04-23T21:22:00.429 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:00.434 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:00.460 INFO:teuthology.orchestra.run.smithi111.stdout:3091 writesame 0xd3deb2e thru 0xd3e8c99 data_size 0x2(0xa16c bytes) 2024-04-23T21:22:00.460 INFO:teuthology.orchestra.run.smithi111.stdout:3092 read 0x67982a3 thru 0x679c6de (0x443c bytes) 2024-04-23T21:22:00.460 INFO:teuthology.orchestra.run.smithi111.stdout:3093 punch from 0x5e90c81 to 0x5e9fcec, (0xf06b bytes) 2024-04-23T21:22:00.460 INFO:teuthology.orchestra.run.smithi111.stdout:3094 write 0x3322000 thru 0x3326c43 (0x4c44 bytes) 2024-04-23T21:22:00.463 INFO:teuthology.orchestra.run.smithi111.stdout:3095 read 0x57f21e thru 0x5800d0 (0xeb3 bytes) 2024-04-23T21:22:00.463 INFO:teuthology.orchestra.run.smithi111.stdout:3097 trunc from 0xd3e8c9a to 0xd2d043c 2024-04-23T21:22:00.468 INFO:teuthology.orchestra.run.smithi111.stdout:3098 writesame 0x1d6f21d thru 0x1d73888 data_size 0x2(0x466c bytes) 2024-04-23T21:22:00.473 INFO:teuthology.orchestra.run.smithi111.stdout:3099 write 0x2336168 thru 0x233f93d (0x97d6 bytes) 2024-04-23T21:22:00.475 INFO:teuthology.orchestra.run.smithi111.stdout:3100 write 0x2db54c thru 0x2ddaa0 (0x2555 bytes) 2024-04-23T21:22:00.476 INFO:teuthology.orchestra.run.smithi111.stdout:3102 writesame 0x5bb0caf thru 0x5bbff31 data_size 0x31(0xf283 bytes) 2024-04-23T21:22:00.480 INFO:teuthology.orchestra.run.smithi111.stdout:3103 writesame 0x90e7939 thru 0x90f0b46 data_size 0x5(0x920e bytes) 2024-04-23T21:22:00.484 INFO:teuthology.orchestra.run.smithi111.stdout:3104 punch from 0x26e949 to 0x26f30e, (0x9c5 bytes) 2024-04-23T21:22:00.484 INFO:teuthology.orchestra.run.smithi111.stdout:3105 read 0x6cec235 thru 0x6cf7bf8 (0xb9c4 bytes) 2024-04-23T21:22:00.484 INFO:teuthology.orchestra.run.smithi111.stdout:3106 read 0x388a90f thru 0x3899453 (0xeb45 bytes) 2024-04-23T21:22:00.484 INFO:teuthology.orchestra.run.smithi111.stdout:3107 write 0x5bf427b thru 0x5bf8707 (0x448d bytes) 2024-04-23T21:22:00.490 INFO:teuthology.orchestra.run.smithi111.stdout:3108 trunc from 0xd2d043c to 0x9126841 2024-04-23T21:22:00.494 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:00.495+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:00.497 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.497 INFO:teuthology.orchestra.run.smithi111.stdout:3110 write 0x8331be4 thru 0x833efa7 (0xd3c4 bytes) 2024-04-23T21:22:00.498 INFO:teuthology.orchestra.run.smithi111.stdout:3111 writesame 0x909353b thru 0x909b331 data_size 0x3(0x7df7 bytes) 2024-04-23T21:22:00.504 INFO:teuthology.orchestra.run.smithi111.stdout:3112 punch from 0x27ba56f to 0x27bf711, (0x51a2 bytes) 2024-04-23T21:22:00.509 INFO:teuthology.orchestra.run.smithi111.stdout:3113 write 0x9292dc5 thru 0x929d207 (0xa443 bytes) 2024-04-23T21:22:00.510 INFO:teuthology.orchestra.run.smithi111.stdout:3114 write 0x6bf6146 thru 0x6bfbaab (0x5966 bytes) 2024-04-23T21:22:00.513 INFO:teuthology.orchestra.run.smithi111.stdout:3115 read 0x6d9c00 thru 0x6dd781 (0x3b82 bytes) 2024-04-23T21:22:00.522 INFO:teuthology.orchestra.run.smithi111.stdout:3116 trunc from 0x929d208 to 0x239b984 2024-04-23T21:22:00.535 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.535 INFO:teuthology.orchestra.run.smithi111.stdout:3118 read 0x1782139 thru 0x178d9a7 (0xb86f bytes) 2024-04-23T21:22:00.536 INFO:teuthology.orchestra.run.smithi111.stdout:3119 read 0x20e1ffd thru 0x20ecb52 (0xab56 bytes) 2024-04-23T21:22:00.541 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:00.541 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:00.541 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:00.541 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:00.541 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:00.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:00.556 INFO:teuthology.orchestra.run.smithi111.stdout:3120 write 0xe5835bf thru 0xe583b34 (0x576 bytes) 2024-04-23T21:22:00.558 INFO:teuthology.orchestra.run.smithi111.stdout:3121 read 0x20a40bb thru 0x20a8679 (0x45bf bytes) 2024-04-23T21:22:00.559 INFO:teuthology.orchestra.run.smithi111.stdout:3122 writesame 0xb43f729 thru 0xb44088d data_size 0x1(0x1165 bytes) 2024-04-23T21:22:00.577 INFO:teuthology.orchestra.run.smithi111.stdout:3123 trunc from 0xe583b35 to 0x2be97ff 2024-04-23T21:22:00.587 INFO:teuthology.orchestra.run.smithi111.stdout:3125 read 0x388dad thru 0x393f4c (0xb1a0 bytes) 2024-04-23T21:22:00.595 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:00.596+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:00.606 INFO:teuthology.orchestra.run.smithi111.stdout:3127 write 0xdd96d12 thru 0xdda4cd8 (0xdfc7 bytes) 2024-04-23T21:22:00.609 INFO:teuthology.orchestra.run.smithi111.stdout:3128 write 0x58e82ee thru 0x58f4199 (0xbeac bytes) 2024-04-23T21:22:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:3130 punch from 0x53dea11 to 0x53eb3ed, (0xc9dc bytes) 2024-04-23T21:22:00.612 INFO:teuthology.orchestra.run.smithi111.stdout:3131 writesame 0xbbd1e1 thru 0xbcd0a7 data_size 0x3(0xfec7 bytes) 2024-04-23T21:22:00.621 INFO:teuthology.orchestra.run.smithi111.stdout:3132 write 0xdbf4d3f thru 0xdbfa260 (0x5522 bytes) 2024-04-23T21:22:00.623 INFO:teuthology.orchestra.run.smithi111.stdout:3133 writesame 0x42c1ff4 thru 0x42c2190 data_size 0x1(0x19d bytes) 2024-04-23T21:22:00.626 INFO:teuthology.orchestra.run.smithi111.stdout:3134 punch from 0x27aa8ba to 0x27b4d7b, (0xa4c1 bytes) 2024-04-23T21:22:00.633 INFO:teuthology.orchestra.run.smithi111.stdout:3135 trunc from 0xdda4cd9 to 0x7e42c02 2024-04-23T21:22:00.640 INFO:teuthology.orchestra.run.smithi111.stdout:3136 write 0x4d232f3 thru 0x4d2b90f (0x861d bytes) 2024-04-23T21:22:00.643 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.644 INFO:teuthology.orchestra.run.smithi111.stdout:3139 trunc from 0x7e42c02 to 0x91f7ea9 2024-04-23T21:22:00.649 INFO:teuthology.orchestra.run.smithi111.stdout:3141 read 0x598d66b thru 0x5997d48 (0xa6de bytes) 2024-04-23T21:22:00.649 INFO:teuthology.orchestra.run.smithi111.stdout:3142 write 0x5ae0621 thru 0x5af034e (0xfd2e bytes) 2024-04-23T21:22:00.651 INFO:teuthology.orchestra.run.smithi111.stdout:3143 read 0x1484e68 thru 0x148b26a (0x6403 bytes) 2024-04-23T21:22:00.660 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:00.660 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:00.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:00.660 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703251999999999, 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:00.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:00.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:00.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.70321388, 2024-04-23T21:22:00.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:00.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.670 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:00.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:00.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:00.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:00.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:00.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:00.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703191480999999, 2024-04-23T21:22:00.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:00.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:00.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:00.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703172085, 2024-04-23T21:22:00.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:00.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:00.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:00.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:00.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:00.699 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703152487000001, 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:00.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:00.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:00.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.705 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703132274, 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703100107999999, 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:00.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703075495, 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:00.715 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703054497, 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:00.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:00.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:00.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:00.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.703034197999999, 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.702956409, 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:00.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:00.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.701461034999999, 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:00.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:00.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:00.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.736 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.701411145, 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.741 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.701387376, 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:00.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.701366370000001, 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.748 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:00.752 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.701346765, 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:00.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:00.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:00.757 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.685344616, 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:00.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:00.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.685326105, 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.762 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:00.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:00.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.685305660000001, 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:00.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:00.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 8.7062284479999992, 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:00.776 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:00.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:00.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:00.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:00.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:00.781 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:00.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3144 write 0xe8496fa thru 0xe852183 (0x8a8a bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3146 write 0x3869217 thru 0x386d9da (0x47c4 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3147 write 0xe5a3006 thru 0xe5b044d (0xd448 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3149 read 0x12fd5b4 thru 0x13086e9 (0xb136 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3150 read 0xde5410f thru 0xde5f25e (0xb150 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3151 write 0x1b5e50a thru 0x1b6c804 (0xe2fb bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3152 read 0x4b1f609 thru 0x4b2e53c (0xef34 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3154 write 0xa9f294 thru 0xaa3efc (0x4c69 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3155 read 0xd5c0a1c thru 0xd5c7af4 (0x70d9 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3156 write 0x966db25 thru 0x9672995 (0x4e71 bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3157 read 0x906371d thru 0x906a107 (0x69eb bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3158 write 0x986cdff thru 0x987389b (0x6a9d bytes) 2024-04-23T21:22:00.787 INFO:teuthology.orchestra.run.smithi111.stdout:3159 read 0x93f8961 thru 0x9406d2e (0xe3ce bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3161 trunc from 0xe852184 to 0x71aab78 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3162 punch from 0x46199fb to 0x46265ff, (0xcc04 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3164 read 0xb60729 thru 0xb68af1 (0x83c9 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3166 write 0xd8fd149 thru 0xd9032f2 (0x61aa bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3167 punch from 0x41a5f12 to 0x41b0769, (0xa857 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3168 writesame 0xe7ab977 thru 0xe7abe6c data_size 0x5(0x4f6 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3169 read 0xaf849f0 thru 0xaf8cd4b (0x835c bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3171 read 0x8145d3 thru 0x81ac40 (0x666e bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3172 punch from 0xc4afb4a to 0xc4bdd52, (0xe208 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3173 write 0xa11ce52 thru 0xa128107 (0xb2b6 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3174 read 0x591f74c thru 0x5929261 (0x9b16 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3175 write 0x83bd398 thru 0x83c8671 (0xb2da bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3176 read 0x2116e84 thru 0x21214ec (0xa669 bytes) 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3177 trunc from 0xe7abe6d to 0x7414fab 2024-04-23T21:22:00.788 INFO:teuthology.orchestra.run.smithi111.stdout:3178 trunc from 0x7414fab to 0xa48230f 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3180 write 0x99afd6e thru 0x99b8622 (0x88b5 bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3182 writesame 0x862439b thru 0x8631809 data_size 0x11(0xd46f bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3183 writesame 0x8cfd568 thru 0x8d001f8 data_size 0x1(0x2c91 bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3184 read 0x223fda4 thru 0x224b6a2 (0xb8ff bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3185 read 0x11bf300 thru 0x11cef11 (0xfc12 bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3186 write 0x3161ba1 thru 0x316744e (0x58ae bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3187 read 0x4df36b5 thru 0x4df9c8f (0x65db bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3188 write 0x456fcd1 thru 0x4577364 (0x7694 bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3189 writesame 0x44751bf thru 0x447882d data_size 0x5(0x366f bytes) 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.789 INFO:teuthology.orchestra.run.smithi111.stdout:3192 write 0xd6bb1bd thru 0xd6c5771 (0xa5b5 bytes) 2024-04-23T21:22:00.790 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:00.783+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:00.790 INFO:teuthology.orchestra.run.smithi111.stdout:3193 punch from 0x7f11ecf to 0x7f16b05, (0x4c36 bytes) 2024-04-23T21:22:00.790 INFO:teuthology.orchestra.run.smithi111.stdout:3194 read 0x4f2b37b thru 0x4f318ba (0x6540 bytes) 2024-04-23T21:22:00.791 INFO:teuthology.orchestra.run.smithi111.stdout:3195 read 0x3669491 thru 0x36705ab (0x711b bytes) 2024-04-23T21:22:00.791 INFO:teuthology.orchestra.run.smithi111.stdout:3196 punch from 0x34f3b17 to 0x3502867, (0xed50 bytes) 2024-04-23T21:22:00.791 INFO:teuthology.orchestra.run.smithi111.stdout:3197 trunc from 0xd6c5772 to 0x980f462 2024-04-23T21:22:00.796 INFO:teuthology.orchestra.run.smithi111.stdout:3198 punch from 0x3d65c4d to 0x3d72a77, (0xce2a bytes) 2024-04-23T21:22:00.797 INFO:teuthology.orchestra.run.smithi111.stdout:3199 punch from 0x44ad2ea to 0x44b1e5a, (0x4b70 bytes) 2024-04-23T21:22:00.802 INFO:teuthology.orchestra.run.smithi111.stdout:3201 trunc from 0x980f462 to 0x4bf4a89 2024-04-23T21:22:00.820 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.820 INFO:teuthology.orchestra.run.smithi111.stdout:3204 punch from 0x24ba190 to 0x24c296c, (0x87dc bytes) 2024-04-23T21:22:00.820 INFO:teuthology.orchestra.run.smithi111.stdout:3205 read 0x27e78c6 thru 0x27eacff (0x343a bytes) 2024-04-23T21:22:00.821 INFO:teuthology.orchestra.run.smithi111.stdout:3206 punch from 0x4a67dcd to 0x4a6ff3c, (0x816f bytes) 2024-04-23T21:22:00.821 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.821 INFO:teuthology.orchestra.run.smithi111.stdout:3208 writesame 0x1c1d46a thru 0x1c25481 data_size 0x4(0x8018 bytes) 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:3209 writesame 0xea3fcfe thru 0xea488e2 data_size 0x1(0x8be5 bytes) 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:3213 punch from 0x7523095 to 0x7528bed, (0x5b58 bytes) 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:3214 punch from 0x98b07 to 0xa544c, (0xc945 bytes) 2024-04-23T21:22:00.842 INFO:teuthology.orchestra.run.smithi111.stdout:3216 write 0x8a94b43 thru 0x8a9b5f8 (0x6ab6 bytes) 2024-04-23T21:22:00.844 INFO:teuthology.orchestra.run.smithi111.stdout:3217 read 0x6077e44 thru 0x607e20d (0x63ca bytes) 2024-04-23T21:22:00.844 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.844 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.844 INFO:teuthology.orchestra.run.smithi111.stdout:3220 write 0xb0e0cbf thru 0xb0ea61b (0x995d bytes) 2024-04-23T21:22:00.845 INFO:teuthology.orchestra.run.smithi111.stdout:3221 read 0xbe23713 thru 0xbe30380 (0xcc6e bytes) 2024-04-23T21:22:00.846 INFO:teuthology.orchestra.run.smithi111.stdout:3224 write 0x60557d7 thru 0x605f7bc (0x9fe6 bytes) 2024-04-23T21:22:00.848 INFO:teuthology.orchestra.run.smithi111.stdout:3225 read 0x730874 thru 0x7329a5 (0x2132 bytes) 2024-04-23T21:22:00.849 INFO:teuthology.orchestra.run.smithi111.stdout:3226 punch from 0xb1b1438 to 0xb1b9878, (0x8440 bytes) 2024-04-23T21:22:00.849 INFO:teuthology.orchestra.run.smithi111.stdout:3227 read 0x9e3fdfe thru 0x9e454ae (0x56b1 bytes) 2024-04-23T21:22:00.849 INFO:teuthology.orchestra.run.smithi111.stdout:3228 punch from 0x75a7f35 to 0x75aa26a, (0x2335 bytes) 2024-04-23T21:22:00.849 INFO:teuthology.orchestra.run.smithi111.stdout:3229 read 0xd2a7b1f thru 0xd2a7ffe (0x4e0 bytes) 2024-04-23T21:22:00.857 INFO:teuthology.orchestra.run.smithi111.stdout:3230 trunc from 0xea488e3 to 0x84cd83a 2024-04-23T21:22:00.866 INFO:teuthology.orchestra.run.smithi111.stdout:3231 punch from 0x4c44b0d to 0x4c466a9, (0x1b9c bytes) 2024-04-23T21:22:00.869 INFO:teuthology.orchestra.run.smithi111.stdout:3233 trunc from 0x84cd83a to 0xa58ad8d 2024-04-23T21:22:00.874 INFO:teuthology.orchestra.run.smithi111.stdout:3234 read 0x7621af3 thru 0x7629646 (0x7b54 bytes) 2024-04-23T21:22:00.879 INFO:teuthology.orchestra.run.smithi111.stdout:3235 writesame 0xac95b64 thru 0xac9a4e9 data_size 0x3(0x4986 bytes) 2024-04-23T21:22:00.880 INFO:teuthology.orchestra.run.smithi111.stdout:3238 read 0x81185d9 thru 0x812158b (0x8fb3 bytes) 2024-04-23T21:22:00.880 INFO:teuthology.orchestra.run.smithi111.stdout:3240 write 0x210615f thru 0x2106b34 (0x9d6 bytes) 2024-04-23T21:22:00.882 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.883 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:00.883+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:00.892 INFO:teuthology.orchestra.run.smithi111.stdout:3245 trunc from 0xac9a4ea to 0x259b58b 2024-04-23T21:22:00.916 INFO:teuthology.orchestra.run.smithi111.stdout:3247 write 0x538b1e5 thru 0x5390d8e (0x5baa bytes) 2024-04-23T21:22:00.919 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.925 INFO:teuthology.orchestra.run.smithi111.stdout:3250 trunc from 0x5390d8f to 0x99c04f2 2024-04-23T21:22:00.930 INFO:teuthology.orchestra.run.smithi111.stdout:3251 punch from 0x1c602b9 to 0x1c6132d, (0x1074 bytes) 2024-04-23T21:22:00.932 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:00.932 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:00.932 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:00.932 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:00.940 INFO:teuthology.orchestra.run.smithi111.stdout:3252 write 0xce27873 thru 0xce2b18c (0x391a bytes) 2024-04-23T21:22:00.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:00.942 INFO:teuthology.orchestra.run.smithi111.stdout:3254 write 0xc0444d3 thru 0xc045c09 (0x1737 bytes) 2024-04-23T21:22:00.944 INFO:teuthology.orchestra.run.smithi111.stdout:3255 read 0x76fb6c2 thru 0x7702e9e (0x77dd bytes) 2024-04-23T21:22:00.952 INFO:teuthology.orchestra.run.smithi111.stdout:3256 writesame 0xe451618 thru 0xe454b24 data_size 0x9(0x350d bytes) 2024-04-23T21:22:00.952 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.952 INFO:teuthology.orchestra.run.smithi111.stdout:3258 writesame 0xd3124d thru 0xd408f7 data_size 0x61(0xf6ab bytes) 2024-04-23T21:22:00.953 INFO:teuthology.orchestra.run.smithi111.stdout:3259 writesame 0x9879bf thru 0x989a8b data_size 0x9(0x20cd bytes) 2024-04-23T21:22:00.955 INFO:teuthology.orchestra.run.smithi111.stdout:3260 read 0xa072fda thru 0xa07ffdc (0xd003 bytes) 2024-04-23T21:22:00.955 INFO:teuthology.orchestra.run.smithi111.stdout:3261 punch from 0xa8ae865 to 0xa8bbf80, (0xd71b bytes) 2024-04-23T21:22:00.955 INFO:teuthology.orchestra.run.smithi111.stdout:3262 writesame 0xbb2b8db thru 0xbb33ef4 data_size 0x5(0x861a bytes) 2024-04-23T21:22:00.961 INFO:teuthology.orchestra.run.smithi111.stdout:3263 punch from 0x2da0dc7 to 0x2da67bd, (0x59f6 bytes) 2024-04-23T21:22:00.961 INFO:teuthology.orchestra.run.smithi111.stdout:3264 punch from 0x351bb13 to 0x3525c97, (0xa184 bytes) 2024-04-23T21:22:00.961 INFO:teuthology.orchestra.run.smithi111.stdout:3265 read 0x6f682b6 thru 0x6f69567 (0x12b2 bytes) 2024-04-23T21:22:00.961 INFO:teuthology.orchestra.run.smithi111.stdout:3266 punch from 0x63c77f7 to 0x63d1689, (0x9e92 bytes) 2024-04-23T21:22:00.962 INFO:teuthology.orchestra.run.smithi111.stdout:3267 read 0x5c37454 thru 0x5c38a66 (0x1613 bytes) 2024-04-23T21:22:00.969 INFO:teuthology.orchestra.run.smithi111.stdout:3268 trunc from 0xe454b25 to 0x707b098 2024-04-23T21:22:00.984 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:00.984+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:00.991 INFO:teuthology.orchestra.run.smithi111.stdout:3270 write 0xe452ab2 thru 0xe46127a (0xe7c9 bytes) 2024-04-23T21:22:00.993 INFO:teuthology.orchestra.run.smithi111.stdout:3271 read 0x3559420 thru 0x3563dbb (0xa99c bytes) 2024-04-23T21:22:00.993 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:00.994 INFO:teuthology.orchestra.run.smithi111.stdout:3273 read 0x73d3633 thru 0x73d658e (0x2f5c bytes) 2024-04-23T21:22:00.994 INFO:teuthology.orchestra.run.smithi111.stdout:3274 write 0xcee3011 thru 0xceec41a (0x940a bytes) 2024-04-23T21:22:00.995 INFO:teuthology.orchestra.run.smithi111.stdout:3276 read 0x2231cec thru 0x2232774 (0xa89 bytes) 2024-04-23T21:22:00.995 INFO:teuthology.orchestra.run.smithi111.stdout:3277 punch from 0x29653ff to 0x2970330, (0xaf31 bytes) 2024-04-23T21:22:00.995 INFO:teuthology.orchestra.run.smithi111.stdout:3278 writesame 0x94b6f1a thru 0x94c40b5 data_size 0xa(0xd19c bytes) 2024-04-23T21:22:00.999 INFO:teuthology.orchestra.run.smithi111.stdout:3279 write 0x3a98a12 thru 0x3aa5c68 (0xd257 bytes) 2024-04-23T21:22:01.002 INFO:teuthology.orchestra.run.smithi111.stdout:3282 punch from 0x5a8d764 to 0x5a93c66, (0x6502 bytes) 2024-04-23T21:22:01.002 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:01.002 INFO:teuthology.orchestra.run.smithi111.stdout:3284 punch from 0x996b706 to 0x99799fb, (0xe2f5 bytes) 2024-04-23T21:22:01.004 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:22:01.007 INFO:teuthology.orchestra.run.smithi111.stdout:3285 writesame 0xe4a0780 thru 0xe4ab931 data_size 0x5(0xb1b2 bytes) 2024-04-23T21:22:01.008 INFO:teuthology.orchestra.run.smithi111.stdout:3287 read 0xcf9b058 thru 0xcf9b37f (0x328 bytes) 2024-04-23T21:22:01.008 INFO:teuthology.orchestra.run.smithi111.stdout:3289 writesame 0xb3b10ae thru 0xb3b7d10 data_size 0x3(0x6c63 bytes) 2024-04-23T21:22:01.013 INFO:teuthology.orchestra.run.smithi111.stdout:3290 write 0x5bdfef thru 0x5c5a98 (0x7aaa bytes) 2024-04-23T21:22:01.015 INFO:teuthology.orchestra.run.smithi111.stdout:3291 punch from 0xe404bf6 to 0xe40e774, (0x9b7e bytes) 2024-04-23T21:22:01.015 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:01.022 INFO:teuthology.orchestra.run.smithi111.stdout:3294 trunc from 0xe4ab932 to 0x7c85d54 2024-04-23T21:22:01.040 INFO:teuthology.orchestra.run.smithi111.stdout:3295 write 0x9eb4b58 thru 0x9ebb884 (0x6d2d bytes) 2024-04-23T21:22:01.042 INFO:teuthology.orchestra.run.smithi111.stdout:3297 read 0x16c0a07 thru 0x16c4b3d (0x4137 bytes) 2024-04-23T21:22:01.042 INFO:teuthology.orchestra.run.smithi111.stdout:3298 read 0x9ad60ab thru 0x9ad765e (0x15b4 bytes) 2024-04-23T21:22:01.043 INFO:teuthology.orchestra.run.smithi111.stdout:3299 write 0x10fabb0 thru 0x10fd6b2 (0x2b03 bytes) 2024-04-23T21:22:01.047 INFO:teuthology.orchestra.run.smithi111.stdout:3300 trunc from 0x9ebb885 to 0xca566fe 2024-04-23T21:22:01.053 INFO:teuthology.orchestra.run.smithi111.stdout:3301 write 0x342492e thru 0x342cbd3 (0x82a6 bytes) 2024-04-23T21:22:01.057 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:01.057 INFO:teuthology.orchestra.run.smithi111.stdout:3303 clone 27 order 25 su 16777216 sc 7 2024-04-23T21:22:01.084 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:01.085+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:01.090 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:01.090 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:01.090 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:01.090 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:01.090 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:01.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:01.184 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:01.185+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959697392999999, 2024-04-23T21:22:01.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:22:01.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:22:01.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959677315, 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.250 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959656686000002, 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:22:01.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:22:01.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:22:01.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:22:01.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959631428000002, 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.255 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959604849000002, 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.259 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:22:01.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:22:01.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959584694999997, 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:22:01.263 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959558934999997, 2024-04-23T21:22:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:22:01.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:22:01.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959527645999998, 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.274 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:01.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:22:01.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:22:01.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959513078999997, 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:22:01.278 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959493663000003, 2024-04-23T21:22:01.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.286 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959478480000001, 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:01.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:22:01.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:22:01.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959467910999997, 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.294 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959458294000001, 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:22:01.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:01.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:22:01.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:22:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959448762000001, 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.299 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.959435763999998, 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:22:01.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:22:01.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.832389806000002, 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.307 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:01.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.83237106, 2024-04-23T21:22:01.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:01.315 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.832335069000003, 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.832307487000001, 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.320 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:22:01.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:22:01.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61147 3.17 3:ef371ac8:::rbd_data.1118e08f6c14.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.85020696799999995, 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037500608999999997, 2024-04-23T21:22:01.324 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61147 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391693+0000", 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412969+0000", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021275891000000002 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412997+0000", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8119999999999998e-05 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413011+0000", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4299e-05 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413101+0000", 2024-04-23T21:22:01.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9747999999999995e-05 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413284+0000", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018242599999999999 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.414592+0000", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001307906 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426984+0000", 2024-04-23T21:22:01.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012392235999999999 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429111+0000", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021271100000000002 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429153+0000", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1839999999999999e-05 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429192+0000", 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9050000000000001e-05 2024-04-23T21:22:01.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.330 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.330 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.330 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:01.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:01.332 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:01.332+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:01.432 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:01.432+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:01.477 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:01.477 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:01.477 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:01.477 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:01.483 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:01.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:01.567 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:01.568+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:01.627 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:01.628 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:01.628 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:01.628 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:01.628 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:01.633 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:01.634+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:01.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:01.734 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:01.735+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:01.777 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:01.777 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:01.781 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126965585000001, 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:01.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:01.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.785 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126944461000001, 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.786 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:01.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:01.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:01.789 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126915194, 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.790 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:01.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:01.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126894432, 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126875406, 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.799 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.802 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126856567000001, 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:01.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:01.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.806 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126823164999999, 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.810 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126803419, 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.811 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126785315999999, 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.815 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126766115000001, 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.819 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:01.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126740716, 2024-04-23T21:22:01.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.827 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126715591, 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:01.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:01.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:01.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.12669058, 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.835 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126667011, 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:01.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:01.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126643855999999, 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.840 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:01.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:01.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126623113000001, 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.844 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:01.848 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126604812, 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:01.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126587232, 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.853 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:01.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 10.126557709, 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:01.857 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:01.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.3873026450000001, 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:01.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:01.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:01.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:01.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:01.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.867 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.868 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:01.868 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:01.868 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:01.870 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-23T21:22:01.601936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1052964176,"num_objects":739,"num_object_clones":200,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":186,"num_read":76486,"num_read_kb":335342,"num_write":9321,"num_write_kb":176910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7069,"ondisk_log_size":7069,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":303344,"kb_used_data":31732,"kb_used_omap":17,"kb_used_meta":271470,"kb_avail":754671376,"statfs":{"total":773094113280,"available":772783489024,"internally_reserved":0,"allocated":32493568,"data_stored":194094947,"data_compressed":10347968,"data_compressed_allocated":11939840,"data_compressed_original":178455146,"omap_allocated":17904,"internal_metadata":277985808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":172637894,"num_objects":120,"num_object_clones":47,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":42,"num_read":6294,"num_read_kb":64357,"num_write":1303,"num_write_kb":31970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.626016"},"pg_stats":[{"pgid":"3.1f","version":"54'214","reported_seq":2456,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.171196+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:21:58.171196+0000","last_peered":"2024-04-23T21:21:58.171196+0000","last_clean":"2024-04-23T21:21:58.171196+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:21:58.171196+0000","last_undegraded":"2024-04-23T21:21:58.171196+0000","last_fullsized":"2024-04-23T21:21:58.171196+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:32.180175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":35796609,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":2639,"num_read_kb":4595,"num_write":284,"num_write_kb":2223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"54'189","reported_seq":3354,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.301367+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:21:58.301367+0000","last_peered":"2024-04-23T21:21:58.301367+0000","last_clean":"2024-04-23T21:21:58.301367+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:21:58.301367+0000","last_undegraded":"2024-04-23T21:21:58.301367+0000","last_fullsized":"2024-04-23T21:21:58.301367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:43:44.714373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":41033334,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":7,"num_read":3365,"num_read_kb":22514,"num_write":215,"num_write_kb":6251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"54'145","reported_seq":3109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.225539+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:21:58.225539+0000","last_peered":"2024-04-23T21:21:58.225539+0000","last_clean":"2024-04-23T21:21:58.225539+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:21:58.225539+0000","last_undegraded":"2024-04-23T21:21:58.225539+0000","last_fullsized":"2024-04-23T21:21:58.225539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:54.356798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":20972537,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3369,"num_read_kb":11849,"num_write":144,"num_write_kb":2190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"54'239","reported_seq":1425,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.261912+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:21:58.261912+0000","last_peered":"2024-04-23T21:21:58.261912+0000","last_clean":"2024-04-23T21:21:58.261912+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:21:58.261912+0000","last_undegraded":"2024-04-23T21:21:58.261912+0000","last_fullsized":"2024-04-23T21:21:58.261912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:41:59.090340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":48918078,"num_objects":36,"num_object_clones":11,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":1492,"num_read_kb":9026,"num_write":310,"num_write_kb":7050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"52'153","reported_seq":5937,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.997892+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:21:57.997892+0000","last_peered":"2024-04-23T21:21:57.997892+0000","last_clean":"2024-04-23T21:21:57.997892+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:21:57.997892+0000","last_undegraded":"2024-04-23T21:21:57.997892+0000","last_fullsized":"2024-04-23T21:21:57.997892+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:45.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":58662735,"num_objects":31,"num_object_clones":12,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":7,"num_read":5742,"num_read_kb":25698,"num_write":187,"num_write_kb":41205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"54'437","reported_seq":2367,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.261132+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:21:58.261132+0000","last_peered":"2024-04-23T21:21:58.261132+0000","last_clean":"2024-04-23T21:21:58.261132+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:21:58.261132+0000","last_undegraded":"2024-04-23T21:21:58.261132+0000","last_fullsized":"2024-04-23T21:21:58.261132+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:26:08.234565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":15064982,"num_objects":24,"num_object_clones":6,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":4129,"num_read_kb":10889,"num_write":660,"num_write_kb":6426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"53'305","reported_seq":1260,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.215357+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:21:57.215357+0000","last_peered":"2024-04-23T21:21:57.215357+0000","last_clean":"2024-04-23T21:21:57.215357+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:21:57.215357+0000","last_undegraded":"2024-04-23T21:21:57.215357+0000","last_fullsized":"2024-04-23T21:21:57.215357+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:13.768955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":28178742,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":2342,"num_read_kb":10646,"num_write":456,"num_write_kb":4763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"54'387","reported_seq":2389,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.255052+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:21:58.255052+0000","last_peered":"2024-04-23T21:21:58.255052+0000","last_clean":"2024-04-23T21:21:58.255052+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:21:58.255052+0000","last_undegraded":"2024-04-23T21:21:58.255052+0000","last_fullsized":"2024-04-23T21:21:58.255052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:22:21.718059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":31812180,"num_objects":32,"num_object_clones":10,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":11,"num_read":4255,"num_read_kb":10110,"num_write":495,"num_write_kb":2002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"54'165","reported_seq":1362,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.088883+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:21:58.088883+0000","last_peered":"2024-04-23T21:21:58.088883+0000","last_clean":"2024-04-23T21:21:58.088883+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:21:58.088883+0000","last_undegraded":"2024-04-23T21:21:58.088883+0000","last_fullsized":"2024-04-23T21:21:58.088883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:21.128574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":23408818,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":2210,"num_read_kb":7318,"num_write":176,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"54'87","reported_seq":767,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.094041+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:21:58.094041+0000","last_peered":"2024-04-23T21:21:58.094041+0000","last_clean":"2024-04-23T21:21:58.094041+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:21:58.094041+0000","last_undegraded":"2024-04-23T21:21:58.094041+0000","last_fullsized":"2024-04-23T21:21:58.094041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:36.244355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":12419798,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":7,"num_read":591,"num_read_kb":6241,"num_write":96,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"54'194","reported_seq":3159,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.212813+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:21:58.212813+0000","last_peered":"2024-04-23T21:21:58.212813+0000","last_clean":"2024-04-23T21:21:58.212813+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:21:58.212813+0000","last_undegraded":"2024-04-23T21:21:58.212813+0000","last_fullsized":"2024-04-23T21:21:58.212813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:14.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":47233735,"num_objects":36,"num_object_clones":12,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":3489,"num_read_kb":11550,"num_write":211,"num_write_kb":5478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"54'186","reported_seq":1242,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.217412+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:21:58.217412+0000","last_peered":"2024-04-23T21:21:58.217412+0000","last_clean":"2024-04-23T21:21:58.217412+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:21:58.217412+0000","last_undegraded":"2024-04-23T21:21:58.217412+0000","last_fullsized":"2024-04-23T21:21:58.217412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:10.000297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":59076106,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1525,"num_read_kb":15622,"num_write":183,"num_write_kb":3960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.043494+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:58.043494+0000","last_peered":"2024-04-23T21:21:58.043494+0000","last_clean":"2024-04-23T21:21:58.043494+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:58.043494+0000","last_undegraded":"2024-04-23T21:21:58.043494+0000","last_fullsized":"2024-04-23T21:21:58.043494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"54'262","reported_seq":882,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.195736+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:21:58.195736+0000","last_peered":"2024-04-23T21:21:58.195736+0000","last_clean":"2024-04-23T21:21:58.195736+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:21:58.195736+0000","last_undegraded":"2024-04-23T21:21:58.195736+0000","last_fullsized":"2024-04-23T21:21:58.195736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:54.828816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":26806942,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1171,"num_read_kb":6700,"num_write":345,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.043533+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:58.043533+0000","last_peered":"2024-04-23T21:21:58.043533+0000","last_clean":"2024-04-23T21:21:58.043533+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:58.043533+0000","last_undegraded":"2024-04-23T21:21:58.043533+0000","last_fullsized":"2024-04-23T21:21:58.043533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"53'136","reported_seq":1012,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.351654+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:21:57.351654+0000","last_peered":"2024-04-23T21:21:57.351654+0000","last_clean":"2024-04-23T21:21:57.351654+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:21:57.351654+0000","last_undegraded":"2024-04-23T21:21:57.351654+0000","last_fullsized":"2024-04-23T21:21:57.351654+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:21:52.380644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":39963241,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":4,"num_read":1030,"num_read_kb":4288,"num_write":133,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.997693+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:57.997693+0000","last_peered":"2024-04-23T21:21:57.997693+0000","last_clean":"2024-04-23T21:21:57.997693+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:57.997693+0000","last_undegraded":"2024-04-23T21:21:57.997693+0000","last_fullsized":"2024-04-23T21:21:57.997693+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"54'217","reported_seq":995,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.228218+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:21:58.228218+0000","last_peered":"2024-04-23T21:21:58.228218+0000","last_clean":"2024-04-23T21:21:58.228218+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:21:58.228218+0000","last_undegraded":"2024-04-23T21:21:58.228218+0000","last_fullsized":"2024-04-23T21:21:58.228218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:50.668055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":31663962,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":1254,"num_read_kb":6940,"num_write":316,"num_write_kb":2443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.000868+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:58.000868+0000","last_peered":"2024-04-23T21:21:58.000868+0000","last_clean":"2024-04-23T21:21:58.000868+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:58.000868+0000","last_undegraded":"2024-04-23T21:21:58.000868+0000","last_fullsized":"2024-04-23T21:21:58.000868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"53'98","reported_seq":139,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977305+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:21:56.977305+0000","last_peered":"2024-04-23T21:21:56.977305+0000","last_clean":"2024-04-23T21:21:56.977305+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:21:56.977305+0000","last_undegraded":"2024-04-23T21:21:56.977305+0000","last_fullsized":"2024-04-23T21:21:56.977305+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:41.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":12145095,"num_objects":18,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":6,"num_read":2150,"num_read_kb":4398,"num_write":113,"num_write_kb":2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"54'107","reported_seq":637,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.230736+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:21:58.230736+0000","last_peered":"2024-04-23T21:21:58.230736+0000","last_clean":"2024-04-23T21:21:58.230736+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:21:58.230736+0000","last_undegraded":"2024-04-23T21:21:58.230736+0000","last_fullsized":"2024-04-23T21:21:58.230736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:37:08.131608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":61336752,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":452,"num_read_kb":8421,"num_write":122,"num_write_kb":3131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993480+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:21:57.993480+0000","last_peered":"2024-04-23T21:21:57.993480+0000","last_clean":"2024-04-23T21:21:57.993480+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:21:57.993480+0000","last_undegraded":"2024-04-23T21:21:57.993480+0000","last_fullsized":"2024-04-23T21:21:57.993480+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":34,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"50'70","reported_seq":354,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:15.468280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":13458230,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2565,"num_read_kb":7402,"num_write":78,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993689+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:57.993689+0000","last_peered":"2024-04-23T21:21:57.993689+0000","last_clean":"2024-04-23T21:21:57.993689+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:57.993689+0000","last_undegraded":"2024-04-23T21:21:57.993689+0000","last_fullsized":"2024-04-23T21:21:57.993689+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"54'495","reported_seq":4296,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.148509+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:21:58.148509+0000","last_peered":"2024-04-23T21:21:58.148509+0000","last_clean":"2024-04-23T21:21:58.148509+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:21:58.148509+0000","last_undegraded":"2024-04-23T21:21:58.148509+0000","last_fullsized":"2024-04-23T21:21:58.148509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:58.277208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":58619986,"num_objects":38,"num_object_clones":12,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":8,"num_read":6515,"num_read_kb":26402,"num_write":779,"num_write_kb":20306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.994207+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:57.994207+0000","last_peered":"2024-04-23T21:21:57.994207+0000","last_clean":"2024-04-23T21:21:57.994207+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:57.994207+0000","last_undegraded":"2024-04-23T21:21:57.994207+0000","last_fullsized":"2024-04-23T21:21:57.994207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"53'331","reported_seq":2788,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.076894+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:21:58.076894+0000","last_peered":"2024-04-23T21:21:58.076894+0000","last_clean":"2024-04-23T21:21:58.076894+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:21:58.076894+0000","last_undegraded":"2024-04-23T21:21:58.076894+0000","last_fullsized":"2024-04-23T21:21:58.076894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:17.005617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":32487175,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":3986,"num_read_kb":7453,"num_write":504,"num_write_kb":2747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993485+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:57.993485+0000","last_peered":"2024-04-23T21:21:57.993485+0000","last_clean":"2024-04-23T21:21:57.993485+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:57.993485+0000","last_undegraded":"2024-04-23T21:21:57.993485+0000","last_fullsized":"2024-04-23T21:21:57.993485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"54'106","reported_seq":882,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.084627+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:21:58.084627+0000","last_peered":"2024-04-23T21:21:58.084627+0000","last_clean":"2024-04-23T21:21:58.084627+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:21:58.084627+0000","last_undegraded":"2024-04-23T21:21:58.084627+0000","last_fullsized":"2024-04-23T21:21:58.084627+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:02.312003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":21043980,"num_objects":23,"num_object_clones":8,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":900,"num_read_kb":11159,"num_write":152,"num_write_kb":8199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"54'155","reported_seq":3105,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.254795+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:21:58.254795+0000","last_peered":"2024-04-23T21:21:58.254795+0000","last_clean":"2024-04-23T21:21:58.254795+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:21:58.254795+0000","last_undegraded":"2024-04-23T21:21:58.254795+0000","last_fullsized":"2024-04-23T21:21:58.254795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:55:17.006627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":46958092,"num_objects":29,"num_object_clones":10,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":8,"num_read":2951,"num_read_kb":17704,"num_write":192,"num_write_kb":6125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"54'257","reported_seq":1518,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.253211+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:21:58.253211+0000","last_peered":"2024-04-23T21:21:58.253211+0000","last_clean":"2024-04-23T21:21:58.253211+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:21:58.253211+0000","last_undegraded":"2024-04-23T21:21:58.253211+0000","last_fullsized":"2024-04-23T21:21:58.253211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:16:16.994467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":19786907,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":4,"num_read":2778,"num_read_kb":8295,"num_write":265,"num_write_kb":4348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"54'131","reported_seq":1013,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.302585+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:21:58.302585+0000","last_peered":"2024-04-23T21:21:58.302585+0000","last_clean":"2024-04-23T21:21:58.302585+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:21:58.302585+0000","last_undegraded":"2024-04-23T21:21:58.302585+0000","last_fullsized":"2024-04-23T21:21:58.302585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:25:08.050695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":26068663,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":1101,"num_read_kb":6618,"num_write":127,"num_write_kb":1079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"54'154","reported_seq":646,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.159440+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:21:58.159440+0000","last_peered":"2024-04-23T21:21:58.159440+0000","last_clean":"2024-04-23T21:21:58.159440+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:21:58.159440+0000","last_undegraded":"2024-04-23T21:21:58.159440+0000","last_fullsized":"2024-04-23T21:21:58.159440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:09.718568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":33331654,"num_objects":21,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":785,"num_read_kb":2845,"num_write":157,"num_write_kb":1987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"54'109","reported_seq":870,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993858+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:21:57.993858+0000","last_peered":"2024-04-23T21:21:57.993858+0000","last_clean":"2024-04-23T21:21:57.993858+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:21:57.993858+0000","last_undegraded":"2024-04-23T21:21:57.993858+0000","last_fullsized":"2024-04-23T21:21:57.993858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:09.173317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":15715007,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":711,"num_read_kb":8571,"num_write":117,"num_write_kb":2032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"54'135","reported_seq":659,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.254616+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:21:58.254616+0000","last_peered":"2024-04-23T21:21:58.254616+0000","last_clean":"2024-04-23T21:21:58.254616+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:21:58.254616+0000","last_undegraded":"2024-04-23T21:21:58.254616+0000","last_fullsized":"2024-04-23T21:21:58.254616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:15:19.181851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":13590028,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1090,"num_read_kb":3517,"num_write":138,"num_write_kb":2745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"54'660","reported_seq":4433,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.251461+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:21:58.251461+0000","last_peered":"2024-04-23T21:21:58.251461+0000","last_clean":"2024-04-23T21:21:58.251461+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:21:58.251461+0000","last_undegraded":"2024-04-23T21:21:58.251461+0000","last_fullsized":"2024-04-23T21:21:58.251461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:41.106618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":47571125,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":7025,"num_read_kb":27431,"num_write":1068,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"54'181","reported_seq":977,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.306332+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:21:58.306332+0000","last_peered":"2024-04-23T21:21:58.306332+0000","last_clean":"2024-04-23T21:21:58.306332+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:21:58.306332+0000","last_undegraded":"2024-04-23T21:21:58.306332+0000","last_fullsized":"2024-04-23T21:21:58.306332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:33:55.932455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":59310575,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":1082,"num_read_kb":13263,"num_write":244,"num_write_kb":6116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"54'101","reported_seq":668,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.224006+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:21:58.224006+0000","last_peered":"2024-04-23T21:21:58.224006+0000","last_clean":"2024-04-23T21:21:58.224006+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:21:58.224006+0000","last_undegraded":"2024-04-23T21:21:58.224006+0000","last_fullsized":"2024-04-23T21:21:58.224006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.303999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":30316704,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":513,"num_read_kb":8384,"num_write":128,"num_write_kb":4868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"54'349","reported_seq":1043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.129530+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:21:58.129530+0000","last_peered":"2024-04-23T21:21:58.129530+0000","last_clean":"2024-04-23T21:21:58.129530+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:21:58.129530+0000","last_undegraded":"2024-04-23T21:21:58.129530+0000","last_fullsized":"2024-04-23T21:21:58.129530+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:59:28.563740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":17591833,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":3,"num_read":2198,"num_read_kb":7406,"num_write":495,"num_write_kb":4557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"54'197","reported_seq":653,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.071086+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:21:58.071086+0000","last_peered":"2024-04-23T21:21:58.071086+0000","last_clean":"2024-04-23T21:21:58.071086+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:21:58.071086+0000","last_undegraded":"2024-04-23T21:21:58.071086+0000","last_fullsized":"2024-04-23T21:21:58.071086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:20:54.897994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":22030203,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":965,"num_read_kb":1987,"num_write":272,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1052373808,"num_objects":737,"num_object_clones":200,"num_object_copies":1474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":186,"num_read":76370,"num_read_kb":335242,"num_write":9162,"num_write_kb":174494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":29229056,"data_stored":202983938,"data_compressed":11059126,"data_compressed_allocated":12902400,"data_compressed_original":188965482,"omap_allocated":0,"internal_metadata":0},"log_size":6952,"ondisk_log_size":6952,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39840,"kb_used_data":4492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332000,"statfs":{"total":96636764160,"available":96595968000,"internally_reserved":0,"allocated":4599808,"data_stored":19801042,"data_compressed":1118546,"data_compressed_allocated":1466368,"data_compressed_original":17375088,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1835008,"data_stored":12060166,"data_compressed":563742,"data_compressed_allocated":565248,"data_compressed_original":11268412,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38600,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333240,"statfs":{"total":96636764160,"available":96597237760,"internally_reserved":0,"allocated":3342336,"data_stored":13363640,"data_compressed":765334,"data_compressed_allocated":983040,"data_compressed_original":11668034,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46388,"kb_used_data":6876,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325452,"statfs":{"total":96636764160,"available":96589262848,"internally_reserved":0,"allocated":7041024,"data_stored":55519760,"data_compressed":2903628,"data_compressed_allocated":3215360,"data_compressed_original":52438353,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40100,"kb_used_data":4764,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331740,"statfs":{"total":96636764160,"available":96595701760,"internally_reserved":0,"allocated":4878336,"data_stored":23723004,"data_compressed":1309049,"data_compressed_allocated":1531904,"data_compressed_original":21092051,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1867776,"data_stored":5923845,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":3152,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":3227648,"data_stored":22809360,"data_compressed":1216014,"data_compressed_allocated":1302528,"data_compressed_original":21346836,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40916,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330924,"statfs":{"total":96636764160,"available":96594866176,"internally_reserved":0,"allocated":5701632,"data_stored":40894130,"data_compressed":2174099,"data_compressed_allocated":2494464,"data_compressed_original":38343897,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5328896,"data_stored":42040344,"data_compressed":2245623,"data_compressed_allocated":2613248,"data_compressed_original":39658713,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2568192,"data_stored":21945182,"data_compressed":1181956,"data_compressed_allocated":1257472,"data_compressed_original":20757012,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1314816,"data_stored":5699601,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4562944,"data_stored":24931924,"data_compressed":1468322,"data_compressed_allocated":1716224,"data_compressed_original":22484691,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6877184,"data_stored":58410382,"data_compressed":3053877,"data_compressed_allocated":3440640,"data_compressed_original":55403857,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3076096,"data_stored":16692432,"data_compressed":948360,"data_compressed_allocated":1249280,"data_compressed_original":15215170,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":11780244,"data_compressed":563742,"data_compressed_allocated":565248,"data_compressed_original":11268412,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":21483829,"data_compressed":1299690,"data_compressed_allocated":1679360,"data_compressed_original":19255152,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:01.872 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-23T21:22:01.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:01.872 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:01.870+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:01.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:01.873 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-23T21:22:01.873 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:22:01.970 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:01.971+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:01.999 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:22:02.019 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:02.019 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:02.019 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:02.019 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:02.028 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:02.066 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone26 intact 2024-04-23T21:22:02.071 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:02.071+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:02.119 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:02.171 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:02.172 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:02.172 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:02.172 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:02.172+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:02.181 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:02.182 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 58, 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:22:02.175528+0000", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:21:48.929598+0000", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:20:59.765723+0000", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5799936, 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:22:02.183 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 5, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "jewel", 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.184 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:02.185 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:02.186 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:02.187 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:02.188 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 2, 2024-04-23T21:22:02.189 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 2, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 1, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:02.190 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:22:02.191 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:02.192 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:02.193 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:02.194 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 8, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 8, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.25, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:22:02.195 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:02.196 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:02.197 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "57", 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:02.198 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 29, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 57, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:02.199 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:02.200 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:02.201 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 1.75, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 1.75, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 1, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.75, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.75, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:02.202 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.04681396484375, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.203 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.204 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:22:02.205 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:22:02.206 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.207 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 13, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 17, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 45, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 48, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:22:02.208 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.209 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.210 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/723341542", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/723341542", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/723341542", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/723341542", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.213 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.218 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.219 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.220 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.221 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.222 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:22:02.223 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.224 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 48, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:22:02.225 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.227 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.229 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 29, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.231 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.232 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:22:02.233 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:02.234 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.235 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:22:02.236 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.239 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 45, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:02.240 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.243 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:22:02.244 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.245 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.247 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.249 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:02.250 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:22:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [], 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [], 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [ 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 2, 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 1 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:02.253 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:02.255 INFO:tasks.thrashosds.thrasher:j is {'epoch': 58, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:22:02.175528+0000', 'last_up_change': '2024-04-23T21:21:48.929598+0000', 'last_in_change': '2024-04-23T21:20:59.765723+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 29, 'snap_epoch': 57, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.04681396484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 17, 'up_from': 45, 'up_thru': 48, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 723341542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 723341542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 723341542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 723341542}]}, 'public_addr': '172.21.15.111:6817/723341542', 'cluster_addr': '172.21.15.111:6819/723341542', 'heartbeat_back_addr': '172.21.15.111:6823/723341542', 'heartbeat_front_addr': '172.21.15.111:6821/723341542', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 45, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:22:02.256 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:02.256 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-04-23T21:22:02.256 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:02.257 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:02.273 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:02.273 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:02.299 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:02.300+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:02.300 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:02.300 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:02.318 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:02.318 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:02.336 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:02.336 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:02.354 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:02.354 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:02.361 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:02.361 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:02.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:02.361 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:02.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3198 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 2981888~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940974933, 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024356404000000002, 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3198 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420555+0000", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420556+0000", 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420558+0000", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420564+0000", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7089999999999999e-06 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420568+0000", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6280000000000004e-06 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444802+0000", 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024234169999999999 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444844+0000", 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1118999999999997e-05 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444912+0000", 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8367999999999993e-05 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940955979999998, 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:02.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:22:02.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.369 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940928616999997, 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:22:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:02.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:22:02.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:22:02.373 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940914368000001, 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.374 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:22:02.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940892124999998, 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940826639000001, 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.382 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.384 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940800897000003, 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.386 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.387 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940776888000002, 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.390 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:02.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:22:02.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940754329000001, 2024-04-23T21:22:02.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:02.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:22:02.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:22:02.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940734526, 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:22:02.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.403 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940715040000001, 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:22:02.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940695241, 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:22:02.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.411 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940675538999997, 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:22:02.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:02.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:22:02.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:22:02.415 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940619413, 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:22:02.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:02.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:22:02.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:22:02.419 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940582192000001, 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.424 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940531878999998, 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:22:02.428 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940498984000001, 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.429 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:22:02.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940476680000003, 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.433 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:22:02.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:22:02.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.940456263999998, 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:22:02.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:22:02.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61148 3.13 3:cc4c0078:::rbd_data.1118e08f6c14.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 1.9687165819999999, 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033335561999999999, 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:02.441 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61148 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392816+0000", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0609999999999998e-06 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407607+0000", 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014790792000000001 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407632+0000", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5092999999999998e-05 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407646+0000", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4134999999999999e-05 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407719+0000", 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3206000000000003e-05 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407831+0000", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111797 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408130+0000", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029870400000000001 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424207+0000", 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016077281999999998 2024-04-23T21:22:02.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426036+0000", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001829378 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426098+0000", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1693999999999996e-05 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426149+0000", 2024-04-23T21:22:02.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1419999999999999e-05 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:02.447 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:02.448 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:02.448 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:02.449 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:02.449 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:02.450 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:02.449+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:02.451 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:02.468 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.468 INFO:teuthology.orchestra.run.smithi111.stdout:3305 writesame 0xa48c30d thru 0xa49c258 data_size 0x2(0xff4c bytes) 2024-04-23T21:22:02.497 INFO:teuthology.orchestra.run.smithi111.stdout:3306 trunc from 0xca566fe to 0x69744a1 2024-04-23T21:22:02.549 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:02.550+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:02.582 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:02.584 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:02.595 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:02.595 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:02.598 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-23T21:22:02.177259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1052964176,"num_objects":739,"num_object_clones":200,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":739,"num_whiteouts":186,"num_read":76486,"num_read_kb":335342,"num_write":9321,"num_write_kb":176910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7069,"ondisk_log_size":7069,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":303344,"kb_used_data":31732,"kb_used_omap":17,"kb_used_meta":271470,"kb_avail":754671376,"statfs":{"total":773094113280,"available":772783489024,"internally_reserved":0,"allocated":32493568,"data_stored":194094947,"data_compressed":10347968,"data_compressed_allocated":11939840,"data_compressed_original":178455146,"omap_allocated":17904,"internal_metadata":277985808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":159031476,"num_objects":113,"num_object_clones":45,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":40,"num_read":6174,"num_read_kb":62073,"num_write":1239,"num_write_kb":28693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.189325"},"pg_stats":[{"pgid":"3.1f","version":"54'214","reported_seq":2456,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.171196+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:21:58.171196+0000","last_peered":"2024-04-23T21:21:58.171196+0000","last_clean":"2024-04-23T21:21:58.171196+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:21:58.171196+0000","last_undegraded":"2024-04-23T21:21:58.171196+0000","last_fullsized":"2024-04-23T21:21:58.171196+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:32.180175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":35796609,"num_objects":13,"num_object_clones":3,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":2639,"num_read_kb":4595,"num_write":284,"num_write_kb":2223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"54'189","reported_seq":3354,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.301367+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:21:58.301367+0000","last_peered":"2024-04-23T21:21:58.301367+0000","last_clean":"2024-04-23T21:21:58.301367+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:21:58.301367+0000","last_undegraded":"2024-04-23T21:21:58.301367+0000","last_fullsized":"2024-04-23T21:21:58.301367+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:43:44.714373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":41033334,"num_objects":28,"num_object_clones":9,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":7,"num_read":3365,"num_read_kb":22514,"num_write":215,"num_write_kb":6251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"54'145","reported_seq":3109,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.225539+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:21:58.225539+0000","last_peered":"2024-04-23T21:21:58.225539+0000","last_clean":"2024-04-23T21:21:58.225539+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:21:58.225539+0000","last_undegraded":"2024-04-23T21:21:58.225539+0000","last_fullsized":"2024-04-23T21:21:58.225539+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:16:54.356798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":20972537,"num_objects":21,"num_object_clones":6,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":3369,"num_read_kb":11849,"num_write":144,"num_write_kb":2190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"54'239","reported_seq":1425,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.261912+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:21:58.261912+0000","last_peered":"2024-04-23T21:21:58.261912+0000","last_clean":"2024-04-23T21:21:58.261912+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:21:58.261912+0000","last_undegraded":"2024-04-23T21:21:58.261912+0000","last_fullsized":"2024-04-23T21:21:58.261912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:41:59.090340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":48918078,"num_objects":36,"num_object_clones":11,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":1492,"num_read_kb":9026,"num_write":310,"num_write_kb":7050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"52'153","reported_seq":5937,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.997892+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:21:57.997892+0000","last_peered":"2024-04-23T21:21:57.997892+0000","last_clean":"2024-04-23T21:21:57.997892+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:21:57.997892+0000","last_undegraded":"2024-04-23T21:21:57.997892+0000","last_fullsized":"2024-04-23T21:21:57.997892+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:49:45.285674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":58662735,"num_objects":31,"num_object_clones":12,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":7,"num_read":5742,"num_read_kb":25698,"num_write":187,"num_write_kb":41205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"54'437","reported_seq":2367,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.261132+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:21:58.261132+0000","last_peered":"2024-04-23T21:21:58.261132+0000","last_clean":"2024-04-23T21:21:58.261132+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:21:58.261132+0000","last_undegraded":"2024-04-23T21:21:58.261132+0000","last_fullsized":"2024-04-23T21:21:58.261132+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:26:08.234565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":15064982,"num_objects":24,"num_object_clones":6,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":6,"num_read":4129,"num_read_kb":10889,"num_write":660,"num_write_kb":6426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"53'305","reported_seq":1260,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.215357+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:21:57.215357+0000","last_peered":"2024-04-23T21:21:57.215357+0000","last_clean":"2024-04-23T21:21:57.215357+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:21:57.215357+0000","last_undegraded":"2024-04-23T21:21:57.215357+0000","last_fullsized":"2024-04-23T21:21:57.215357+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:45:13.768955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":28178742,"num_objects":23,"num_object_clones":6,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":7,"num_read":2342,"num_read_kb":10646,"num_write":456,"num_write_kb":4763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"54'387","reported_seq":2389,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.255052+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:21:58.255052+0000","last_peered":"2024-04-23T21:21:58.255052+0000","last_clean":"2024-04-23T21:21:58.255052+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:21:58.255052+0000","last_undegraded":"2024-04-23T21:21:58.255052+0000","last_fullsized":"2024-04-23T21:21:58.255052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:22:21.718059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":31812180,"num_objects":32,"num_object_clones":10,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":11,"num_read":4255,"num_read_kb":10110,"num_write":495,"num_write_kb":2002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"54'165","reported_seq":1362,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.088883+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:21:58.088883+0000","last_peered":"2024-04-23T21:21:58.088883+0000","last_clean":"2024-04-23T21:21:58.088883+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:21:58.088883+0000","last_undegraded":"2024-04-23T21:21:58.088883+0000","last_fullsized":"2024-04-23T21:21:58.088883+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:00:21.128574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":23408818,"num_objects":25,"num_object_clones":5,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":7,"num_read":2210,"num_read_kb":7318,"num_write":176,"num_write_kb":5532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"54'87","reported_seq":767,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.094041+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:21:58.094041+0000","last_peered":"2024-04-23T21:21:58.094041+0000","last_clean":"2024-04-23T21:21:58.094041+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:21:58.094041+0000","last_undegraded":"2024-04-23T21:21:58.094041+0000","last_fullsized":"2024-04-23T21:21:58.094041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:36.244355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":12419798,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":7,"num_read":591,"num_read_kb":6241,"num_write":96,"num_write_kb":2352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"54'194","reported_seq":3159,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.212813+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:21:58.212813+0000","last_peered":"2024-04-23T21:21:58.212813+0000","last_clean":"2024-04-23T21:21:58.212813+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:21:58.212813+0000","last_undegraded":"2024-04-23T21:21:58.212813+0000","last_fullsized":"2024-04-23T21:21:58.212813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:43:14.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":47233735,"num_objects":36,"num_object_clones":12,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":11,"num_read":3489,"num_read_kb":11550,"num_write":211,"num_write_kb":5478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"54'186","reported_seq":1242,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.217412+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:21:58.217412+0000","last_peered":"2024-04-23T21:21:58.217412+0000","last_clean":"2024-04-23T21:21:58.217412+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:21:58.217412+0000","last_undegraded":"2024-04-23T21:21:58.217412+0000","last_fullsized":"2024-04-23T21:21:58.217412+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:19:10.000297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":59076106,"num_objects":26,"num_object_clones":8,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1525,"num_read_kb":15622,"num_write":183,"num_write_kb":3960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.043494+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:21:58.043494+0000","last_peered":"2024-04-23T21:21:58.043494+0000","last_clean":"2024-04-23T21:21:58.043494+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:21:58.043494+0000","last_undegraded":"2024-04-23T21:21:58.043494+0000","last_fullsized":"2024-04-23T21:21:58.043494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"54'262","reported_seq":882,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.195736+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:21:58.195736+0000","last_peered":"2024-04-23T21:21:58.195736+0000","last_clean":"2024-04-23T21:21:58.195736+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:21:58.195736+0000","last_undegraded":"2024-04-23T21:21:58.195736+0000","last_fullsized":"2024-04-23T21:21:58.195736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:14:54.828816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":26806942,"num_objects":26,"num_object_clones":6,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":7,"num_read":1171,"num_read_kb":6700,"num_write":345,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.043533+0000","last_change":"2024-04-23T21:21:09.823788+0000","last_active":"2024-04-23T21:21:58.043533+0000","last_peered":"2024-04-23T21:21:58.043533+0000","last_clean":"2024-04-23T21:21:58.043533+0000","last_became_active":"2024-04-23T21:21:09.823443+0000","last_became_peered":"2024-04-23T21:21:09.823443+0000","last_unstale":"2024-04-23T21:21:58.043533+0000","last_undegraded":"2024-04-23T21:21:58.043533+0000","last_fullsized":"2024-04-23T21:21:58.043533+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.409781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"53'136","reported_seq":1012,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.351654+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:21:57.351654+0000","last_peered":"2024-04-23T21:21:57.351654+0000","last_clean":"2024-04-23T21:21:57.351654+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:21:57.351654+0000","last_undegraded":"2024-04-23T21:21:57.351654+0000","last_fullsized":"2024-04-23T21:21:57.351654+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:21:52.380644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":39963241,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":4,"num_read":1030,"num_read_kb":4288,"num_write":133,"num_write_kb":1111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.997693+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:21:57.997693+0000","last_peered":"2024-04-23T21:21:57.997693+0000","last_clean":"2024-04-23T21:21:57.997693+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:21:57.997693+0000","last_undegraded":"2024-04-23T21:21:57.997693+0000","last_fullsized":"2024-04-23T21:21:57.997693+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"54'217","reported_seq":995,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.228218+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:21:58.228218+0000","last_peered":"2024-04-23T21:21:58.228218+0000","last_clean":"2024-04-23T21:21:58.228218+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:21:58.228218+0000","last_undegraded":"2024-04-23T21:21:58.228218+0000","last_fullsized":"2024-04-23T21:21:58.228218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:50.668055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":31663962,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":1254,"num_read_kb":6940,"num_write":316,"num_write_kb":2443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.000868+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:21:58.000868+0000","last_peered":"2024-04-23T21:21:58.000868+0000","last_clean":"2024-04-23T21:21:58.000868+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:21:58.000868+0000","last_undegraded":"2024-04-23T21:21:58.000868+0000","last_fullsized":"2024-04-23T21:21:58.000868+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"53'98","reported_seq":139,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977305+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:21:56.977305+0000","last_peered":"2024-04-23T21:21:56.977305+0000","last_clean":"2024-04-23T21:21:56.977305+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:21:56.977305+0000","last_undegraded":"2024-04-23T21:21:56.977305+0000","last_fullsized":"2024-04-23T21:21:56.977305+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:41.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":12145095,"num_objects":18,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":6,"num_read":2150,"num_read_kb":4398,"num_write":113,"num_write_kb":2292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1","version":"54'107","reported_seq":637,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.230736+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:21:58.230736+0000","last_peered":"2024-04-23T21:21:58.230736+0000","last_clean":"2024-04-23T21:21:58.230736+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:21:58.230736+0000","last_undegraded":"2024-04-23T21:21:58.230736+0000","last_fullsized":"2024-04-23T21:21:58.230736+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:37:08.131608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":61336752,"num_objects":23,"num_object_clones":7,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":452,"num_read_kb":8421,"num_write":122,"num_write_kb":3131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":243,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993480+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:21:57.993480+0000","last_peered":"2024-04-23T21:21:57.993480+0000","last_clean":"2024-04-23T21:21:57.993480+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:21:57.993480+0000","last_undegraded":"2024-04-23T21:21:57.993480+0000","last_fullsized":"2024-04-23T21:21:57.993480+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":34,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"50'70","reported_seq":354,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-04-23T21:21:56.977137+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:21:56.977137+0000","last_peered":"2024-04-23T21:21:56.977137+0000","last_clean":"2024-04-23T21:21:56.977137+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:21:56.977137+0000","last_undegraded":"2024-04-23T21:21:56.977137+0000","last_fullsized":"2024-04-23T21:21:56.977137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:15.468280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":13458230,"num_objects":17,"num_object_clones":4,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":2565,"num_read_kb":7402,"num_write":78,"num_write_kb":1354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993689+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:21:57.993689+0000","last_peered":"2024-04-23T21:21:57.993689+0000","last_clean":"2024-04-23T21:21:57.993689+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:21:57.993689+0000","last_undegraded":"2024-04-23T21:21:57.993689+0000","last_fullsized":"2024-04-23T21:21:57.993689+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"54'495","reported_seq":4296,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.148509+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:21:58.148509+0000","last_peered":"2024-04-23T21:21:58.148509+0000","last_clean":"2024-04-23T21:21:58.148509+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:21:58.148509+0000","last_undegraded":"2024-04-23T21:21:58.148509+0000","last_fullsized":"2024-04-23T21:21:58.148509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:26:58.277208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":58619986,"num_objects":38,"num_object_clones":12,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":8,"num_read":6515,"num_read_kb":26402,"num_write":779,"num_write_kb":20306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.994207+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:21:57.994207+0000","last_peered":"2024-04-23T21:21:57.994207+0000","last_clean":"2024-04-23T21:21:57.994207+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:21:57.994207+0000","last_undegraded":"2024-04-23T21:21:57.994207+0000","last_fullsized":"2024-04-23T21:21:57.994207+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"53'331","reported_seq":2788,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.076894+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:21:58.076894+0000","last_peered":"2024-04-23T21:21:58.076894+0000","last_clean":"2024-04-23T21:21:58.076894+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:21:58.076894+0000","last_undegraded":"2024-04-23T21:21:58.076894+0000","last_fullsized":"2024-04-23T21:21:58.076894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:17.005617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":32487175,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":4,"num_read":3986,"num_read_kb":7453,"num_write":504,"num_write_kb":2747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":88,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993485+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:21:57.993485+0000","last_peered":"2024-04-23T21:21:57.993485+0000","last_clean":"2024-04-23T21:21:57.993485+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:21:57.993485+0000","last_undegraded":"2024-04-23T21:21:57.993485+0000","last_fullsized":"2024-04-23T21:21:57.993485+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"54'106","reported_seq":882,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.084627+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:21:58.084627+0000","last_peered":"2024-04-23T21:21:58.084627+0000","last_clean":"2024-04-23T21:21:58.084627+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:21:58.084627+0000","last_undegraded":"2024-04-23T21:21:58.084627+0000","last_fullsized":"2024-04-23T21:21:58.084627+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:33:02.312003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":21043980,"num_objects":23,"num_object_clones":8,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":900,"num_read_kb":11159,"num_write":152,"num_write_kb":8199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"54'155","reported_seq":3105,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.254795+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:21:58.254795+0000","last_peered":"2024-04-23T21:21:58.254795+0000","last_clean":"2024-04-23T21:21:58.254795+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:21:58.254795+0000","last_undegraded":"2024-04-23T21:21:58.254795+0000","last_fullsized":"2024-04-23T21:21:58.254795+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:55:17.006627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":46958092,"num_objects":29,"num_object_clones":10,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":8,"num_read":2951,"num_read_kb":17704,"num_write":192,"num_write_kb":6125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"54'257","reported_seq":1518,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.253211+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:21:58.253211+0000","last_peered":"2024-04-23T21:21:58.253211+0000","last_clean":"2024-04-23T21:21:58.253211+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:21:58.253211+0000","last_undegraded":"2024-04-23T21:21:58.253211+0000","last_fullsized":"2024-04-23T21:21:58.253211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:16:16.994467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":19786907,"num_objects":20,"num_object_clones":5,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":4,"num_read":2778,"num_read_kb":8295,"num_write":265,"num_write_kb":4348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"54'131","reported_seq":1013,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.302585+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:21:58.302585+0000","last_peered":"2024-04-23T21:21:58.302585+0000","last_clean":"2024-04-23T21:21:58.302585+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:21:58.302585+0000","last_undegraded":"2024-04-23T21:21:58.302585+0000","last_fullsized":"2024-04-23T21:21:58.302585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:25:08.050695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":26068663,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":1101,"num_read_kb":6618,"num_write":127,"num_write_kb":1079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"54'154","reported_seq":646,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.159440+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:21:58.159440+0000","last_peered":"2024-04-23T21:21:58.159440+0000","last_clean":"2024-04-23T21:21:58.159440+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:21:58.159440+0000","last_undegraded":"2024-04-23T21:21:58.159440+0000","last_fullsized":"2024-04-23T21:21:58.159440+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:09.718568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":33331654,"num_objects":21,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":6,"num_read":785,"num_read_kb":2845,"num_write":157,"num_write_kb":1987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"54'109","reported_seq":870,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:57.993858+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:21:57.993858+0000","last_peered":"2024-04-23T21:21:57.993858+0000","last_clean":"2024-04-23T21:21:57.993858+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:21:57.993858+0000","last_undegraded":"2024-04-23T21:21:57.993858+0000","last_fullsized":"2024-04-23T21:21:57.993858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:48:09.173317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":15715007,"num_objects":16,"num_object_clones":5,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":711,"num_read_kb":8571,"num_write":117,"num_write_kb":2032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"54'135","reported_seq":659,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.254616+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:21:58.254616+0000","last_peered":"2024-04-23T21:21:58.254616+0000","last_clean":"2024-04-23T21:21:58.254616+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:21:58.254616+0000","last_undegraded":"2024-04-23T21:21:58.254616+0000","last_fullsized":"2024-04-23T21:21:58.254616+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:15:19.181851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":13590028,"num_objects":10,"num_object_clones":1,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1090,"num_read_kb":3517,"num_write":138,"num_write_kb":2745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"54'660","reported_seq":4433,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.251461+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:21:58.251461+0000","last_peered":"2024-04-23T21:21:58.251461+0000","last_clean":"2024-04-23T21:21:58.251461+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:21:58.251461+0000","last_undegraded":"2024-04-23T21:21:58.251461+0000","last_fullsized":"2024-04-23T21:21:58.251461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:14:41.106618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":47571125,"num_objects":26,"num_object_clones":4,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":5,"num_read":7025,"num_read_kb":27431,"num_write":1068,"num_write_kb":5499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"54'181","reported_seq":977,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.306332+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:21:58.306332+0000","last_peered":"2024-04-23T21:21:58.306332+0000","last_clean":"2024-04-23T21:21:58.306332+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:21:58.306332+0000","last_undegraded":"2024-04-23T21:21:58.306332+0000","last_fullsized":"2024-04-23T21:21:58.306332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:33:55.932455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":59310575,"num_objects":19,"num_object_clones":2,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":1082,"num_read_kb":13263,"num_write":244,"num_write_kb":6116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"54'101","reported_seq":668,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.224006+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:21:58.224006+0000","last_peered":"2024-04-23T21:21:58.224006+0000","last_clean":"2024-04-23T21:21:58.224006+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:21:58.224006+0000","last_undegraded":"2024-04-23T21:21:58.224006+0000","last_fullsized":"2024-04-23T21:21:58.224006+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:46.303999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":30316704,"num_objects":19,"num_object_clones":5,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":513,"num_read_kb":8384,"num_write":128,"num_write_kb":4868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"54'349","reported_seq":1043,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.129530+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:21:58.129530+0000","last_peered":"2024-04-23T21:21:58.129530+0000","last_clean":"2024-04-23T21:21:58.129530+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:21:58.129530+0000","last_undegraded":"2024-04-23T21:21:58.129530+0000","last_fullsized":"2024-04-23T21:21:58.129530+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:59:28.563740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":17591833,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":3,"num_read":2198,"num_read_kb":7406,"num_write":495,"num_write_kb":4557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"54'197","reported_seq":653,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-23T21:21:58.071086+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:21:58.071086+0000","last_peered":"2024-04-23T21:21:58.071086+0000","last_clean":"2024-04-23T21:21:58.071086+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:21:58.071086+0000","last_undegraded":"2024-04-23T21:21:58.071086+0000","last_fullsized":"2024-04-23T21:21:58.071086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:20:54.897994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":22030203,"num_objects":23,"num_object_clones":5,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":965,"num_read_kb":1987,"num_write":272,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1052373808,"num_objects":737,"num_object_clones":200,"num_object_copies":1474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":186,"num_read":76370,"num_read_kb":335242,"num_write":9162,"num_write_kb":174494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":29229056,"data_stored":202983938,"data_compressed":11059126,"data_compressed_allocated":12902400,"data_compressed_original":188965482,"omap_allocated":0,"internal_metadata":0},"log_size":6952,"ondisk_log_size":6952,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39840,"kb_used_data":4492,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332000,"statfs":{"total":96636764160,"available":96595968000,"internally_reserved":0,"allocated":4599808,"data_stored":19801042,"data_compressed":1118546,"data_compressed_allocated":1466368,"data_compressed_original":17375088,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1835008,"data_stored":12060166,"data_compressed":563742,"data_compressed_allocated":565248,"data_compressed_original":11268412,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38600,"kb_used_data":3264,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333240,"statfs":{"total":96636764160,"available":96597237760,"internally_reserved":0,"allocated":3342336,"data_stored":13363640,"data_compressed":765334,"data_compressed_allocated":983040,"data_compressed_original":11668034,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574861,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46388,"kb_used_data":6876,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94325452,"statfs":{"total":96636764160,"available":96589262848,"internally_reserved":0,"allocated":7041024,"data_stored":55519760,"data_compressed":2903628,"data_compressed_allocated":3215360,"data_compressed_original":52438353,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574861,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40100,"kb_used_data":4764,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331740,"statfs":{"total":96636764160,"available":96595701760,"internally_reserved":0,"allocated":4878336,"data_stored":23723004,"data_compressed":1309049,"data_compressed_allocated":1531904,"data_compressed_original":21092051,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1867776,"data_stored":5923845,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528323,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":3152,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":3227648,"data_stored":22809360,"data_compressed":1216014,"data_compressed_allocated":1302528,"data_compressed_original":21346836,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40916,"kb_used_data":5568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330924,"statfs":{"total":96636764160,"available":96594866176,"internally_reserved":0,"allocated":5701632,"data_stored":40894130,"data_compressed":2174099,"data_compressed_allocated":2494464,"data_compressed_original":38343897,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5328896,"data_stored":42040344,"data_compressed":2245623,"data_compressed_allocated":2613248,"data_compressed_original":39658713,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2568192,"data_stored":21945182,"data_compressed":1181956,"data_compressed_allocated":1257472,"data_compressed_original":20757012,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1314816,"data_stored":5699601,"data_compressed":297556,"data_compressed_allocated":380928,"data_compressed_original":4922475,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4562944,"data_stored":24931924,"data_compressed":1468322,"data_compressed_allocated":1716224,"data_compressed_original":22484691,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":6877184,"data_stored":58410382,"data_compressed":3053877,"data_compressed_allocated":3440640,"data_compressed_original":55403857,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3076096,"data_stored":16692432,"data_compressed":948360,"data_compressed_allocated":1249280,"data_compressed_original":15215170,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":11780244,"data_compressed":563742,"data_compressed_allocated":565248,"data_compressed_original":11268412,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4288512,"data_stored":21483829,"data_compressed":1299690,"data_compressed_allocated":1679360,"data_compressed_original":19255152,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:02.600 INFO:tasks.thrashosds.thrasher:Setting 1.6 to [2, 0, 6, 7] 2024-04-23T21:22:02.600 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.6', '2', '0', '6', '7'] 2024-04-23T21:22:02.600 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 1.6 2 0 6 7 2024-04-23T21:22:02.601 INFO:teuthology.orchestra.run.smithi111.stdout:3307 read 0x18708f0 thru 0x18780e9 (0x77fa bytes) 2024-04-23T21:22:02.602 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.602 INFO:teuthology.orchestra.run.smithi111.stdout:3309 write 0x23ad507 thru 0x23b08b6 (0x33b0 bytes) 2024-04-23T21:22:02.604 INFO:teuthology.orchestra.run.smithi111.stdout:3310 write 0x1c8efec thru 0x1c98c02 (0x9c17 bytes) 2024-04-23T21:22:02.605 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:02.606 INFO:teuthology.orchestra.run.smithi111.stdout:3311 punch from 0x1519828 to 0x151b53f, (0x1d17 bytes) 2024-04-23T21:22:02.617 INFO:teuthology.orchestra.run.smithi111.stdout:3314 write 0xc97547e thru 0xc982a8e (0xd611 bytes) 2024-04-23T21:22:02.619 INFO:teuthology.orchestra.run.smithi111.stdout:3315 writesame 0xb576444 thru 0xb5817c0 data_size 0x1(0xb37d bytes) 2024-04-23T21:22:02.640 INFO:teuthology.orchestra.run.smithi111.stdout:3316 writesame 0x829b4e4 thru 0x82a4789 data_size 0x3(0x92a6 bytes) 2024-04-23T21:22:02.645 INFO:teuthology.orchestra.run.smithi111.stdout:3319 read 0x2131245 thru 0x2131bdd (0x999 bytes) 2024-04-23T21:22:02.646 INFO:teuthology.orchestra.run.smithi111.stdout:3320 writesame 0x84a1b6e thru 0x84ad1a0 data_size 0x1(0xb633 bytes) 2024-04-23T21:22:02.650 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:02.651+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:02.668 INFO:teuthology.orchestra.run.smithi111.stdout:3321 read 0x29ee9a3 thru 0x29f405f (0x56bd bytes) 2024-04-23T21:22:02.678 INFO:teuthology.orchestra.run.smithi111.stdout:3322 trunc from 0xc982a8f to 0x54cb9ef 2024-04-23T21:22:02.687 INFO:teuthology.orchestra.run.smithi111.stdout:3323 punch from 0x4c1f701 to 0x4c2313b, (0x3a3a bytes) 2024-04-23T21:22:02.687 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.687 INFO:teuthology.orchestra.run.smithi111.stdout:3326 write 0xb7df47 thru 0xb8669b (0x8755 bytes) 2024-04-23T21:22:02.689 INFO:teuthology.orchestra.run.smithi111.stdout:3328 read 0x9b5a1a thru 0x9bcf16 (0x74fd bytes) 2024-04-23T21:22:02.689 INFO:teuthology.orchestra.run.smithi111.stdout:3332 read 0x5323d49 thru 0x532877e (0x4a36 bytes) 2024-04-23T21:22:02.690 INFO:teuthology.orchestra.run.smithi111.stdout:3333 writesame 0x1897e87 thru 0x189b758 data_size 0x7(0x38d2 bytes) 2024-04-23T21:22:02.702 INFO:teuthology.orchestra.run.smithi111.stdout:3334 write 0x97b6008 thru 0x97c5785 (0xf77e bytes) 2024-04-23T21:22:02.713 INFO:teuthology.orchestra.run.smithi111.stdout:3335 write 0xcd6b2b1 thru 0xcd779f8 (0xc748 bytes) 2024-04-23T21:22:02.714 INFO:teuthology.orchestra.run.smithi111.stdout:3336 punch from 0xa8c58f1 to 0xa8d4ea6, (0xf5b5 bytes) 2024-04-23T21:22:02.714 INFO:teuthology.orchestra.run.smithi111.stdout:3337 read 0x310f8e4 thru 0x3115431 (0x5b4e bytes) 2024-04-23T21:22:02.715 INFO:teuthology.orchestra.run.smithi111.stdout:3338 read 0x3cf3928 thru 0x3cfa39f (0x6a78 bytes) 2024-04-23T21:22:02.715 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.715 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.715 INFO:teuthology.orchestra.run.smithi111.stdout:3343 writesame 0x29ab390 thru 0x29b0f61 data_size 0x49(0x5bd2 bytes) 2024-04-23T21:22:02.718 INFO:teuthology.orchestra.run.smithi111.stdout:3345 trunc from 0xcd779f9 to 0xdf9123e 2024-04-23T21:22:02.724 INFO:teuthology.orchestra.run.smithi111.stdout:3346 write 0xa6449b4 thru 0xa64d044 (0x8691 bytes) 2024-04-23T21:22:02.725 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:02.725 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:02.725 INFO:teuthology.orchestra.run.smithi111.stdout:3348 writesame 0x360f65 thru 0x363697 data_size 0xf(0x2733 bytes) 2024-04-23T21:22:02.727 INFO:teuthology.orchestra.run.smithi111.stdout:3349 writesame 0x38c45a0 thru 0x38d06ee data_size 0x29(0xc14f bytes) 2024-04-23T21:22:02.728 INFO:teuthology.orchestra.run.smithi111.stdout:3352 read 0xa3ac6 thru 0xb0b6b (0xd0a6 bytes) 2024-04-23T21:22:02.729 INFO:teuthology.orchestra.run.smithi111.stdout:3353 write 0xcf0ce37 thru 0xcf19a29 (0xcbf3 bytes) 2024-04-23T21:22:02.731 INFO:teuthology.orchestra.run.smithi111.stdout:3354 write 0x7e0d14a thru 0x7e12613 (0x54ca bytes) 2024-04-23T21:22:02.733 INFO:teuthology.orchestra.run.smithi111.stdout:3355 write 0xc0c5e30 thru 0xc0c6851 (0xa22 bytes) 2024-04-23T21:22:02.734 INFO:teuthology.orchestra.run.smithi111.stdout:3356 punch from 0x390af76 to 0x3919066, (0xe0f0 bytes) 2024-04-23T21:22:02.736 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:02.749 INFO:teuthology.orchestra.run.smithi111.stdout:3357 trunc from 0xdf9123e to 0x1d4619c 2024-04-23T21:22:02.751 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:02.752+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:02.772 INFO:teuthology.orchestra.run.smithi111.stdout:3358 trunc from 0x1d4619c to 0xccaf51 2024-04-23T21:22:02.783 INFO:teuthology.orchestra.run.smithi111.stdout:3359 trunc from 0xccaf51 to 0x13a2b19 2024-04-23T21:22:02.788 INFO:teuthology.orchestra.run.smithi111.stdout:3360 punch from 0x2fdb11 to 0x309fb9, (0xc4a8 bytes) 2024-04-23T21:22:02.813 INFO:teuthology.orchestra.run.smithi111.stdout:3361 write 0xddbca5e thru 0xddc1c08 (0x51ab bytes) 2024-04-23T21:22:02.817 INFO:teuthology.orchestra.run.smithi111.stdout:3362 write 0x5795b48 thru 0x57a183a (0xbcf3 bytes) 2024-04-23T21:22:02.821 INFO:teuthology.orchestra.run.smithi111.stdout:3363 read 0x27c5539 thru 0x27caab0 (0x5578 bytes) 2024-04-23T21:22:02.821 INFO:teuthology.orchestra.run.smithi111.stdout:3364 write 0xa111c23 thru 0xa11a38d (0x876b bytes) 2024-04-23T21:22:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.824 INFO:teuthology.orchestra.run.smithi111.stdout:3366 write 0x7d42430 thru 0x7d456df (0x32b0 bytes) 2024-04-23T21:22:02.825 INFO:teuthology.orchestra.run.smithi111.stdout:3367 writesame 0x9086c28 thru 0x9091ef5 data_size 0x9(0xb2ce bytes) 2024-04-23T21:22:02.830 INFO:teuthology.orchestra.run.smithi111.stdout:3370 writesame 0x1f12721 thru 0x1f1f2b0 data_size 0x8(0xcb90 bytes) 2024-04-23T21:22:02.835 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:02.836 INFO:teuthology.orchestra.run.smithi111.stdout:3374 writesame 0xf8509f thru 0xf921da data_size 0xe(0xd13c bytes) 2024-04-23T21:22:02.838 INFO:teuthology.orchestra.run.smithi111.stdout:3375 write 0x4f92264 thru 0x4f9e590 (0xc32d bytes) 2024-04-23T21:22:02.849 INFO:teuthology.orchestra.run.smithi111.stdout:3376 trunc from 0xddc1c09 to 0x75c25b5 2024-04-23T21:22:02.852 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:02.853+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:02.856 INFO:teuthology.orchestra.run.smithi111.stdout:3379 clone 28 order 24 su 4194304 sc 13 2024-04-23T21:22:02.862 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:02.862 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:02.871 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:02.954 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:02.954+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:03.015 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.016 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.024 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:03.054 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:03.055+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:03.127 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.128 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:03.155 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:03.156+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:03.180 INFO:teuthology.orchestra.run.smithi111.stderr:set 1.6 pg_upmap_items mapping to [2->0,6->7] 2024-04-23T21:22:03.246 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone27 from 0x75c25b5 (overlap 0xccaf51) to 0xa3bf74 2024-04-23T21:22:03.249 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.249 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.256 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:03.257+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:03.260 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:03.322 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-04-23T21:22:03.357 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:03.358+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:03.417 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.417 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:03.458 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:03.459+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:03.559 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:03.560+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:03.566 INFO:teuthology.orchestra.run.smithi111.stdout:3382 write 0x288a376 thru 0x288a472 (0xfd bytes) 2024-04-23T21:22:03.573 INFO:teuthology.orchestra.run.smithi111.stdout:3383 write 0xcb16da thru 0xcbdecb (0xc7f2 bytes) 2024-04-23T21:22:03.575 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.576 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.582 INFO:teuthology.orchestra.run.smithi111.stdout:3384 read 0x52ddc97 thru 0x52e90b5 (0xb41f bytes) 2024-04-23T21:22:03.585 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:03.594 INFO:teuthology.orchestra.run.smithi111.stdout:3385 write 0xc3eba77 thru 0xc3f8ec1 (0xd44b bytes) 2024-04-23T21:22:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:3386 read 0x53a61c3 thru 0x53b2b86 (0xc9c4 bytes) 2024-04-23T21:22:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.597 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.598 INFO:teuthology.orchestra.run.smithi111.stdout:3391 writesame 0x1548f03 thru 0x1552d46 data_size 0xe(0x9e44 bytes) 2024-04-23T21:22:03.602 INFO:teuthology.orchestra.run.smithi111.stdout:3392 read 0x5fb4d92 thru 0x5fc23fe (0xd66d bytes) 2024-04-23T21:22:03.602 INFO:teuthology.orchestra.run.smithi111.stdout:3393 write 0x851a7c2 thru 0x8521c45 (0x7484 bytes) 2024-04-23T21:22:03.606 INFO:teuthology.orchestra.run.smithi111.stdout:3395 writesame 0x4416cc0 thru 0x441d578 data_size 0x13(0x68b9 bytes) 2024-04-23T21:22:03.609 INFO:teuthology.orchestra.run.smithi111.stdout:3396 write 0x73dd72c thru 0x73e2831 (0x5106 bytes) 2024-04-23T21:22:03.632 INFO:teuthology.orchestra.run.smithi111.stdout:3398 writesame 0x5b8ed00 thru 0x5b91141 data_size 0x4e(0x2442 bytes) 2024-04-23T21:22:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:3400 read 0x3fe4e52 thru 0x3fec3f5 (0x75a4 bytes) 2024-04-23T21:22:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:3401 read 0x25cc7fe thru 0x25d3468 (0x6c6b bytes) 2024-04-23T21:22:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:3403 read 0x3f8cd78 thru 0x3f90e90 (0x4119 bytes) 2024-04-23T21:22:03.635 INFO:teuthology.orchestra.run.smithi111.stdout:3404 read 0x285f9a6 thru 0x286e8a0 (0xeefb bytes) 2024-04-23T21:22:03.636 INFO:teuthology.orchestra.run.smithi111.stdout:3405 write 0xb8b0dbf thru 0xb8b8183 (0x73c5 bytes) 2024-04-23T21:22:03.637 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.637 INFO:teuthology.orchestra.run.smithi111.stdout:3408 read 0x74c1ce8 thru 0x74d1502 (0xf81b bytes) 2024-04-23T21:22:03.638 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.638 INFO:teuthology.orchestra.run.smithi111.stdout:3410 write 0x5062a1e thru 0x5065fb7 (0x359a bytes) 2024-04-23T21:22:03.639 INFO:teuthology.orchestra.run.smithi111.stdout:3411 read 0x664f81b thru 0x665ed74 (0xf55a bytes) 2024-04-23T21:22:03.639 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.639 INFO:teuthology.orchestra.run.smithi111.stdout:3414 read 0x19e68d3 thru 0x19efd13 (0x9441 bytes) 2024-04-23T21:22:03.640 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.640 INFO:teuthology.orchestra.run.smithi111.stdout:3416 writesame 0x4b42021 thru 0x4b5029f data_size 0x1(0xe27f bytes) 2024-04-23T21:22:03.660 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:03.661+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:03.666 INFO:teuthology.orchestra.run.smithi111.stdout:3417 read 0x4d3d412 thru 0x4d4788d (0xa47c bytes) 2024-04-23T21:22:03.667 INFO:teuthology.orchestra.run.smithi111.stdout:3419 write 0x6621dc thru 0x668a36 (0x685b bytes) 2024-04-23T21:22:03.689 INFO:teuthology.orchestra.run.smithi111.stdout:3420 writesame 0x741dc14 thru 0x74290c0 data_size 0x1(0xb4ad bytes) 2024-04-23T21:22:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.700 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.709 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:03.712 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.714 INFO:teuthology.orchestra.run.smithi111.stdout:3423 trunc from 0xc3f8ec2 to 0x9d5a0b1 2024-04-23T21:22:03.743 INFO:teuthology.orchestra.run.smithi111.stdout:3424 punch from 0x12ba9e9 to 0x12be90e, (0x3f25 bytes) 2024-04-23T21:22:03.743 INFO:teuthology.orchestra.run.smithi111.stdout:3425 writesame 0x4e3d18a thru 0x4e418cb data_size 0x7(0x4742 bytes) 2024-04-23T21:22:03.744 INFO:teuthology.orchestra.run.smithi111.stdout:3426 punch from 0x8a824b1 to 0x8a8f146, (0xcc95 bytes) 2024-04-23T21:22:03.751 INFO:teuthology.orchestra.run.smithi111.stdout:3427 writesame 0xb1856cc thru 0xb19326b data_size 0x2(0xdba0 bytes) 2024-04-23T21:22:03.761 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:03.762+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:03.761 INFO:teuthology.orchestra.run.smithi111.stdout:3429 writesame 0xe59810a thru 0xe59847c data_size 0x1(0x373 bytes) 2024-04-23T21:22:03.761 INFO:teuthology.orchestra.run.smithi111.stdout:3430 read 0x3e93de9 thru 0x3e9db8d (0x9da5 bytes) 2024-04-23T21:22:03.771 INFO:teuthology.orchestra.run.smithi111.stdout:3431 trunc from 0xe59847d to 0x6673cbd 2024-04-23T21:22:03.782 INFO:teuthology.orchestra.run.smithi111.stdout:3432 write 0x15e59ee thru 0x15f584f (0xfe62 bytes) 2024-04-23T21:22:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:3433 read 0x2bbdd68 thru 0x2bc9043 (0xb2dc bytes) 2024-04-23T21:22:03.796 INFO:teuthology.orchestra.run.smithi111.stdout:3434 write 0xbbf82db thru 0xbc04fb6 (0xccdc bytes) 2024-04-23T21:22:03.797 INFO:teuthology.orchestra.run.smithi111.stdout:3435 punch from 0x6c49ae to 0x6c553f, (0xb91 bytes) 2024-04-23T21:22:03.797 INFO:teuthology.orchestra.run.smithi111.stdout:3436 writesame 0x5510c0c thru 0x5515c65 data_size 0x16(0x505a bytes) 2024-04-23T21:22:03.798 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.799 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.799 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:03.799 INFO:teuthology.orchestra.run.smithi111.stdout:3441 writesame 0x2304ab0 thru 0x23069d0 data_size 0x1(0x1f21 bytes) 2024-04-23T21:22:03.803 INFO:teuthology.orchestra.run.smithi111.stdout:3444 writesame 0xa057709 thru 0xa05da00 data_size 0x4(0x62f8 bytes) 2024-04-23T21:22:03.806 INFO:teuthology.orchestra.run.smithi111.stdout:3445 punch from 0x557231 to 0x5658d0, (0xe69f bytes) 2024-04-23T21:22:03.806 INFO:teuthology.orchestra.run.smithi111.stdout:3446 write 0x60f7116 thru 0x60ff76f (0x865a bytes) 2024-04-23T21:22:03.808 INFO:teuthology.orchestra.run.smithi111.stdout:3447 clone 29 order 21 su 131072 sc 7 2024-04-23T21:22:03.824 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.824 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.833 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:03.862 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:03.863+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:03.963 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:03.964+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:03.975 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:03.976 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:03.984 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:04.014 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:22:04.064 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:04.065+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:04.130 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:04.130 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:04.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:04.165 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:04.166+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:04.251 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:04.251 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:04.258 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone28 intact 2024-04-23T21:22:04.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:04.266 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:04.267+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:04.363 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:04.363 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:04.367 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:04.368+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:04.371 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:04.435 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-04-23T21:22:04.452 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:04.469 INFO:teuthology.orchestra.run.smithi111.stdout:3448 write 0x49ee1d thru 0x4a1592 (0x2776 bytes) 2024-04-23T21:22:04.469 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:04.470+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:04.481 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:04.481 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:04.487 INFO:teuthology.orchestra.run.smithi111.stdout:3449 read 0xba99bac thru 0xbaa00d6 (0x652b bytes) 2024-04-23T21:22:04.488 INFO:teuthology.orchestra.run.smithi111.stdout:3450 writesame 0xa23ea65 thru 0xa24e9ca data_size 0x33(0xff66 bytes) 2024-04-23T21:22:04.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:04.494 INFO:teuthology.orchestra.run.smithi111.stdout:3451 read 0x26dbc5e thru 0x26eae70 (0xf213 bytes) 2024-04-23T21:22:04.495 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.495 INFO:teuthology.orchestra.run.smithi111.stdout:3453 punch from 0x7635587 to 0x7636139, (0xbb2 bytes) 2024-04-23T21:22:04.495 INFO:teuthology.orchestra.run.smithi111.stdout:3454 writesame 0x15aca3e thru 0x15ade42 data_size 0x5(0x1405 bytes) 2024-04-23T21:22:04.500 INFO:teuthology.orchestra.run.smithi111.stdout:3456 write 0xaf9d85e thru 0xafa5795 (0x7f38 bytes) 2024-04-23T21:22:04.516 INFO:teuthology.orchestra.run.smithi111.stdout:3457 write 0xe5a11af thru 0xe5aaa47 (0x9899 bytes) 2024-04-23T21:22:04.520 INFO:teuthology.orchestra.run.smithi111.stdout:3459 write 0xb8eba0f thru 0xb8efc56 (0x4248 bytes) 2024-04-23T21:22:04.524 INFO:teuthology.orchestra.run.smithi111.stdout:3463 read 0xcf0a01d thru 0xcf12542 (0x8526 bytes) 2024-04-23T21:22:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:3465 read 0x32e3c63 thru 0x32f31ae (0xf54c bytes) 2024-04-23T21:22:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:3466 read 0xc46bcc6 thru 0xc4750d4 (0x940f bytes) 2024-04-23T21:22:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:3467 read 0x1ac762d thru 0x1ad2ba6 (0xb57a bytes) 2024-04-23T21:22:04.525 INFO:teuthology.orchestra.run.smithi111.stdout:3468 writesame 0xc7fe2e3 thru 0xc7ff4ba data_size 0x4(0x11d8 bytes) 2024-04-23T21:22:04.540 INFO:teuthology.orchestra.run.smithi111.stdout:3470 trunc from 0xe5aaa48 to 0x528fe0d 2024-04-23T21:22:04.568 INFO:teuthology.orchestra.run.smithi111.stdout:3471 write 0x39f71f8 thru 0x39fc492 (0x529b bytes) 2024-04-23T21:22:04.569 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:04.569+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:04.572 INFO:teuthology.orchestra.run.smithi111.stdout:3472 write 0x1af4489 thru 0x1b02ff1 (0xeb69 bytes) 2024-04-23T21:22:04.577 INFO:teuthology.orchestra.run.smithi111.stdout:3473 read 0x3cfa1e3 thru 0x3d077c6 (0xd5e4 bytes) 2024-04-23T21:22:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:04.584 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:04.584 INFO:teuthology.orchestra.run.smithi111.stdout:3475 write 0x5dcbd17 thru 0x5ddbc32 (0xff1c bytes) 2024-04-23T21:22:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:3479 read 0x1ea593 thru 0x1f5bcf (0xb63d bytes) 2024-04-23T21:22:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:3480 punch from 0x15280 to 0x1ff29, (0xaca9 bytes) 2024-04-23T21:22:04.588 INFO:teuthology.orchestra.run.smithi111.stdout:3481 read 0x1d1f735 thru 0x1d2ee40 (0xf70c bytes) 2024-04-23T21:22:04.591 INFO:teuthology.orchestra.run.smithi111.stdout:3483 trunc from 0x5ddbc33 to 0x812e021 2024-04-23T21:22:04.602 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:04.603 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:04.607 INFO:teuthology.orchestra.run.smithi111.stdout:3484 write 0xb69c73b thru 0xb6aa992 (0xe258 bytes) 2024-04-23T21:22:04.608 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:04.608 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:04.611 INFO:teuthology.orchestra.run.smithi111.stdout:3485 read 0x8771e35 thru 0x87813d1 (0xf59d bytes) 2024-04-23T21:22:04.611 INFO:teuthology.orchestra.run.smithi111.stdout:3489 write 0x5f7068e thru 0x5f7192a (0x129d bytes) 2024-04-23T21:22:04.613 INFO:teuthology.orchestra.run.smithi111.stdout:3490 read 0xa1df57d thru 0xa1ee15e (0xebe2 bytes) 2024-04-23T21:22:04.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:04.622 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:04.622 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:04.625 INFO:teuthology.orchestra.run.smithi111.stdout:3492 trunc from 0xb6aa993 to 0x1d66d46 2024-04-23T21:22:04.642 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:04.643 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:04.649 INFO:teuthology.orchestra.run.smithi111.stdout:3494 punch from 0x1028536 to 0x102f178, (0x6c42 bytes) 2024-04-23T21:22:04.661 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:04.661 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:04.668 INFO:teuthology.orchestra.run.smithi111.stdout:3495 writesame 0xd36c335 thru 0xd373550 data_size 0x43(0x721c bytes) 2024-04-23T21:22:04.668 INFO:teuthology.orchestra.run.smithi111.stdout:3496 read 0x9230471 thru 0x9238798 (0x8328 bytes) 2024-04-23T21:22:04.669 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:04.670+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:04.674 INFO:teuthology.orchestra.run.smithi111.stdout:3497 write 0xe46ea5f thru 0xe47c51f (0xdac1 bytes) 2024-04-23T21:22:04.677 INFO:teuthology.orchestra.run.smithi111.stdout:3499 writesame 0xa2b9b29 thru 0xa2bec8f data_size 0xd(0x5167 bytes) 2024-04-23T21:22:04.679 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:04.679 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:04.681 INFO:teuthology.orchestra.run.smithi111.stdout:3501 punch from 0x5894ad8 to 0x5899511, (0x4a39 bytes) 2024-04-23T21:22:04.681 INFO:teuthology.orchestra.run.smithi111.stdout:3502 write 0x562a80d thru 0x563531e (0xab12 bytes) 2024-04-23T21:22:04.693 INFO:teuthology.orchestra.run.smithi111.stdout:3503 trunc from 0xe47c520 to 0x65c48dd 2024-04-23T21:22:04.698 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:04.698 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:04.716 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:04.716 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:04.718 INFO:teuthology.orchestra.run.smithi111.stdout:3504 write 0xda1a51a thru 0xda23de6 (0x98cd bytes) 2024-04-23T21:22:04.722 INFO:teuthology.orchestra.run.smithi111.stdout:3505 punch from 0x10891db to 0x1094ac5, (0xb8ea bytes) 2024-04-23T21:22:04.722 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.722 INFO:teuthology.orchestra.run.smithi111.stdout:3507 writesame 0x829a10c thru 0x829a372 data_size 0x5(0x267 bytes) 2024-04-23T21:22:04.725 INFO:teuthology.orchestra.run.smithi111.stdout:3510 writesame 0xbdd2112 thru 0xbde1767 data_size 0x1(0xf656 bytes) 2024-04-23T21:22:04.755 INFO:teuthology.orchestra.run.smithi111.stdout:3511 read 0xd0272cf thru 0xd028f5b (0x1c8d bytes) 2024-04-23T21:22:04.755 INFO:teuthology.orchestra.run.smithi111.stdout:3512 read 0x7c03665 thru 0x7c11b2e (0xe4ca bytes) 2024-04-23T21:22:04.755 INFO:teuthology.orchestra.run.smithi111.stdout:3513 read 0xa1d4ada thru 0xa1dae29 (0x6350 bytes) 2024-04-23T21:22:04.755 INFO:teuthology.orchestra.run.smithi111.stdout:3514 writesame 0x3c18bef thru 0x3c1db78 data_size 0x1(0x4f8a bytes) 2024-04-23T21:22:04.765 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.765 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.769 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:04.769 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:04.769 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:04.769 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:04.769 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:04.770 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:04.770+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:04.772 INFO:teuthology.orchestra.run.smithi111.stdout:3517 write 0xeda32aa thru 0xeda5267 (0x1fbe bytes) 2024-04-23T21:22:04.775 INFO:teuthology.orchestra.run.smithi111.stdout:3519 punch from 0xe322721 to 0xe32679d, (0x407c bytes) 2024-04-23T21:22:04.775 INFO:teuthology.orchestra.run.smithi111.stdout:3520 write 0xa12106c thru 0xa1291be (0x8153 bytes) 2024-04-23T21:22:04.778 INFO:teuthology.orchestra.run.smithi111.stdout:3521 read 0x4502214 thru 0x450d161 (0xaf4e bytes) 2024-04-23T21:22:04.778 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.779 INFO:teuthology.orchestra.run.smithi111.stdout:3523 punch from 0x58c0ff9 to 0x58cfc37, (0xec3e bytes) 2024-04-23T21:22:04.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:04.794 INFO:teuthology.orchestra.run.smithi111.stdout:3524 trunc from 0xeda5268 to 0x1627bb4 2024-04-23T21:22:04.836 INFO:teuthology.orchestra.run.smithi111.stdout:3527 write 0xc2b4e50 thru 0xc2c4962 (0xfb13 bytes) 2024-04-23T21:22:04.842 INFO:teuthology.orchestra.run.smithi111.stdout:3528 trunc from 0xc2c4963 to 0xbbf09af 2024-04-23T21:22:04.849 INFO:teuthology.orchestra.run.smithi111.stdout:3529 read 0x68c755 thru 0x6987b5 (0xc061 bytes) 2024-04-23T21:22:04.850 INFO:teuthology.orchestra.run.smithi111.stdout:3530 read 0x1729a0b thru 0x1730297 (0x688d bytes) 2024-04-23T21:22:04.851 INFO:teuthology.orchestra.run.smithi111.stdout:3531 writesame 0x83ebd26 thru 0x83f2c35 data_size 0x8(0x6f10 bytes) 2024-04-23T21:22:04.854 INFO:teuthology.orchestra.run.smithi111.stdout:3533 read 0x17ae10e thru 0x17af8f1 (0x17e4 bytes) 2024-04-23T21:22:04.855 INFO:teuthology.orchestra.run.smithi111.stdout:3535 write 0x11d5759 thru 0x11e1c61 (0xc509 bytes) 2024-04-23T21:22:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:3536 read 0x74bbae0 thru 0x74c17b7 (0x5cd8 bytes) 2024-04-23T21:22:04.856 INFO:teuthology.orchestra.run.smithi111.stdout:3537 read 0x60df005 thru 0x60e3062 (0x405e bytes) 2024-04-23T21:22:04.857 INFO:teuthology.orchestra.run.smithi111.stdout:3538 write 0x834a835 thru 0x835068d (0x5e59 bytes) 2024-04-23T21:22:04.859 INFO:teuthology.orchestra.run.smithi111.stdout:3539 punch from 0xa8ccb59 to 0xa8d679f, (0x9c46 bytes) 2024-04-23T21:22:04.868 INFO:teuthology.orchestra.run.smithi111.stdout:3540 write 0xda64558 thru 0xda73adc (0xf585 bytes) 2024-04-23T21:22:04.870 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:04.871+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:04.871 INFO:teuthology.orchestra.run.smithi111.stdout:3541 writesame 0xbf9b4a7 thru 0xbfa3e0a data_size 0x12(0x8964 bytes) 2024-04-23T21:22:04.877 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:04.877 INFO:teuthology.orchestra.run.smithi111.stdout:3543 write 0x71ede8f thru 0x71f4f58 (0x70ca bytes) 2024-04-23T21:22:04.881 INFO:teuthology.orchestra.run.smithi111.stdout:3544 read 0xaca1b81 thru 0xacad05b (0xb4db bytes) 2024-04-23T21:22:04.881 INFO:teuthology.orchestra.run.smithi111.stdout:3546 write 0x8b25bf5 thru 0x8b2cb19 (0x6f25 bytes) 2024-04-23T21:22:04.882 INFO:teuthology.orchestra.run.smithi111.stdout:3547 write 0xaf22f8a thru 0xaf2a5f4 (0x766b bytes) 2024-04-23T21:22:04.887 INFO:teuthology.orchestra.run.smithi111.stdout:3550 write 0x53b0d42 thru 0x53bf486 (0xe745 bytes) 2024-04-23T21:22:04.891 INFO:teuthology.orchestra.run.smithi111.stdout:3551 writesame 0x302bb56 thru 0x30318b2 data_size 0x1f(0x5d5d bytes) 2024-04-23T21:22:04.894 INFO:teuthology.orchestra.run.smithi111.stdout:3552 write 0xb560283 thru 0xb56fd3c (0xfaba bytes) 2024-04-23T21:22:04.898 INFO:teuthology.orchestra.run.smithi111.stdout:3553 read 0xc47c87f thru 0xc4861af (0x9931 bytes) 2024-04-23T21:22:04.903 INFO:teuthology.orchestra.run.smithi111.stdout:3554 trunc from 0xda73add to 0x994b5e8 2024-04-23T21:22:04.913 INFO:teuthology.orchestra.run.smithi111.stdout:3555 punch from 0x3e291c to 0x3e62ef, (0x39d3 bytes) 2024-04-23T21:22:04.913 INFO:teuthology.orchestra.run.smithi111.stdout:3556 write 0xcb5028 thru 0xcb5c8f (0xc68 bytes) 2024-04-23T21:22:04.924 INFO:teuthology.orchestra.run.smithi111.stdout:3557 punch from 0x58e292a to 0x58eb874, (0x8f4a bytes) 2024-04-23T21:22:04.932 INFO:teuthology.orchestra.run.smithi111.stdout:3558 write 0xb44c9b7 thru 0xb44f5e0 (0x2c2a bytes) 2024-04-23T21:22:04.935 INFO:teuthology.orchestra.run.smithi111.stdout:3559 writesame 0x92fd7c4 thru 0x9309369 data_size 0x1(0xbba6 bytes) 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2141 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7438336~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654755745999999, 2024-04-23T21:22:04.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032180034000000003, 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2141 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:04.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282201+0000", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282203+0000", 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:22:04.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282209+0000", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9209999999999997e-06 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282214+0000", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9049999999999996e-06 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314304+0000", 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032090081999999999 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314341+0000", 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7338000000000001e-05 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314381+0000", 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9289000000000003e-05 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2142 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7503872~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.941 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654735668000001, 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032340164999999997, 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2142 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282221+0000", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282223+0000", 2024-04-23T21:22:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.928e-06 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282230+0000", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2269999999999996e-06 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282235+0000", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314413+0000", 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032177641999999999 2024-04-23T21:22:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314485+0000", 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2279000000000004e-05 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314561+0000", 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5919000000000003e-05 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2143 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7569408~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654715039000003, 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.03242225, 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.946 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2143 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282241+0000", 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282242+0000", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282249+0000", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.968e-06 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282255+0000", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4609999999999996e-06 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282260+0000", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5449999999999997e-06 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314588+0000", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032327896000000002 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314625+0000", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6992000000000001e-05 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314664+0000", 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9017000000000003e-05 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2144 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7634944~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654689781000002, 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032576807999999999, 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2144 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282267+0000", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282275+0000", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1920000000000005e-06 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282281+0000", 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282286+0000", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9810000000000003e-06 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314695+0000", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032408758000000003 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314770+0000", 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5251000000000002e-05 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314844+0000", 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3046000000000005e-05 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2145 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7700480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654663202000002, 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032659090000000002, 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.954 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2145 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282293+0000", 2024-04-23T21:22:04.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282294+0000", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282295+0000", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7999999999999999e-06 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282302+0000", 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9770000000000003e-06 2024-04-23T21:22:04.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282307+0000", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8740000000000003e-06 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314870+0000", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032562672000000001 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314907+0000", 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7014000000000002e-05 2024-04-23T21:22:04.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314952+0000", 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5343000000000003e-05 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2146 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7766016~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654643047999997, 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032820347999999999, 2024-04-23T21:22:04.958 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2146 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282313+0000", 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282314+0000", 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282316+0000", 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.781e-06 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282322+0000", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6329999999999999e-06 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282332+0000", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0275000000000001e-05 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.314984+0000", 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032651769999999997 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315060+0000", 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5273999999999998e-05 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315134+0000", 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2147 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7831552~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.963 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654617287999997, 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032909002, 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2147 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:04.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282339+0000", 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282340+0000", 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282341+0000", 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9489999999999999e-06 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282347+0000", 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8889999999999999e-06 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282352+0000", 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315161+0000", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032809302999999998 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315201+0000", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9541e-05 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315248+0000", 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7511e-05 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.967 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2148 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7897088~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654585998999998, 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033063618000000003, 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.968 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2148 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282370+0000", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282371+0000", 2024-04-23T21:22:04.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5499999999999999e-07 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282372+0000", 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.39e-06 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282378+0000", 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7740000000000001e-06 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282381+0000", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9749999999999999e-06 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315281+0000", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032899928000000002 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315358+0000", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7674000000000002e-05 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315434+0000", 2024-04-23T21:22:04.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5622000000000002e-05 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2149 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 7962624~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654571431999997, 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033152418000000003, 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2149 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282385+0000", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0900000000000001e-07 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282386+0000", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.119e-06 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282390+0000", 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282392+0000", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4310000000000001e-06 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315461+0000", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033069130000000002 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315499+0000", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8006000000000002e-05 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315537+0000", 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8291999999999999e-05 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2150 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8028160~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654552015999997, 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033313078000000003, 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2150 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282404+0000", 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282405+0000", 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282406+0000", 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.155e-06 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282409+0000", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3139999999999999e-06 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282416+0000", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315570+0000", 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033153683000000003 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315645+0000", 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5051999999999999e-05 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315718+0000", 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2781999999999995e-05 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2151 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8093696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654536833000002, 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033402414999999998, 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2151 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282420+0000", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999998e-07 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282421+0000", 2024-04-23T21:22:04.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1260000000000001e-06 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282424+0000", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.106e-06 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282427+0000", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315744+0000", 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033317055999999998 2024-04-23T21:22:04.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315781+0000", 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7196000000000001e-05 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315822+0000", 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1291000000000002e-05 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2152 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8159232~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654526263999998, 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033574960000000001, 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2152 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282430+0000", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282431+0000", 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8000000000000004e-07 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282434+0000", 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0699999999999998e-06 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282437+0000", 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3879999999999998e-06 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315853+0000", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033416567000000001 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.315931+0000", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7812000000000006e-05 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316005+0000", 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4021000000000002e-05 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2153 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8224768~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654516647000001, 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033670653000000002, 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2153 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282440+0000", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282441+0000", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8299999999999995e-07 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282444+0000", 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9969999999999999e-06 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282446+0000", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316033+0000", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033587073000000002 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316071+0000", 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7409e-05 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316110+0000", 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9620999999999999e-05 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2154 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8290304~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654507115000001, 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033840666999999998, 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.993 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2154 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282449+0000", 2024-04-23T21:22:04.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282450+0000", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.22e-07 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282453+0000", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6119999999999999e-06 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282456+0000", 2024-04-23T21:22:04.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.287e-06 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282459+0000", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5399999999999998e-06 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316142+0000", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033682777999999997 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316218+0000", 2024-04-23T21:22:04.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6042000000000001e-05 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316290+0000", 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2186000000000003e-05 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:2155 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 8355840~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.654494116999999, 2024-04-23T21:22:04.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033920646999999998, 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 2155 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:04.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282462+0000", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282463+0000", 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.05e-07 2024-04-23T21:22:04.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282464+0000", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0279999999999999e-06 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282467+0000", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0319999999999999e-06 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.282472+0000", 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7640000000000002e-06 2024-04-23T21:22:05.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316305+0000", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033832866000000003 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316345+0000", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9311999999999999e-05 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.316383+0000", 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8439999999999998e-05 2024-04-23T21:22:05.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3083 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.527448159000002, 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032103549000000002, 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3083 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409508+0000", 2024-04-23T21:22:05.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409509+0000", 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409511+0000", 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8309999999999999e-06 2024-04-23T21:22:05.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409517+0000", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409521+0000", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4039999999999997e-06 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441533+0000", 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032011684999999998 2024-04-23T21:22:05.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441571+0000", 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7988999999999998e-05 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441612+0000", 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1124000000000001e-05 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3084 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 3997696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.527429413, 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032204618999999997, 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3084 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409527+0000", 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7599999999999998e-07 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409529+0000", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8649999999999999e-06 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409535+0000", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7289999999999997e-06 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409539+0000", 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441635+0000", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032095802 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441689+0000", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4236e-05 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441732+0000", 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2345999999999997e-05 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.010 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3085 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4063232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.527393422000003, 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032271266999999999, 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3085 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.011 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409563+0000", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409564+0000", 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1799999999999995e-07 2024-04-23T21:22:05.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409566+0000", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409572+0000", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1650000000000003e-06 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409583+0000", 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1544e-05 2024-04-23T21:22:05.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441756+0000", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032172266999999997 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441791+0000", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5735000000000003e-05 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441834+0000", 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2842000000000001e-05 2024-04-23T21:22:05.014 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3086 3.1d 3:b8ee1a78:::rbd_data.1118c9f76dd8.0000000000000005:4 [read 4128768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 40.527365840000002, 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032355812999999997, 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.015 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3086 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409591+0000", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9800000000000004e-07 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409600+0000", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0699999999999996e-06 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409610+0000", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5580000000000006e-06 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.409620+0000", 2024-04-23T21:22:05.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0676000000000001e-05 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441849+0000", 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.032228538000000001 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441885+0000", 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5583999999999997e-05 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.441946+0000", 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1889000000000003e-05 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61147 3.17 3:ef371ac8:::rbd_data.1118e08f6c14.0000000000000001:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.5452653210000005, 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037500608999999997, 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.020 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61147 2024-04-23T21:22:05.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.020 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391691+0000", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.391693+0000", 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:05.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412969+0000", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021275891000000002 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412997+0000", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8119999999999998e-05 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413011+0000", 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4299e-05 2024-04-23T21:22:05.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413101+0000", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9747999999999995e-05 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413284+0000", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018242599999999999 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.414592+0000", 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001307906 2024-04-23T21:22:05.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426984+0000", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012392235999999999 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429111+0000", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0021271100000000002 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429153+0000", 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1839999999999999e-05 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.429192+0000", 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9050000000000001e-05 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.025 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.027 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:05.028 INFO:teuthology.orchestra.run.smithi111.stdout:3560 punch from 0x40b0b22 to 0x40bcaff, (0xbfdd bytes) 2024-04-23T21:22:05.028 INFO:teuthology.orchestra.run.smithi111.stdout:3561 write 0xbb25c37 thru 0xbb3223b (0xc605 bytes) 2024-04-23T21:22:05.028 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.028 INFO:teuthology.orchestra.run.smithi111.stdout:3563 trunc from 0xbb3223c to 0x68d0dec 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3564 write 0x65ffa5f thru 0x6605744 (0x5ce6 bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3565 read 0x6148603 thru 0x614969b (0x1099 bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3567 punch from 0x19206a4 to 0x1925d10, (0x566c bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3569 write 0x40b8212 thru 0x40bebe3 (0x69d2 bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3570 punch from 0x9b02a4 to 0x9b86e0, (0x843c bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3571 writesame 0xcf4aeda thru 0xcf54aac data_size 0x1(0x9bd3 bytes) 2024-04-23T21:22:05.029 INFO:teuthology.orchestra.run.smithi111.stdout:3572 trunc from 0xcf54aad to 0x52973a8 2024-04-23T21:22:05.029 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:05.027+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:05.030 INFO:teuthology.orchestra.run.smithi111.stdout:3574 writesame 0x8ce78f2 thru 0x8cee867 data_size 0x2(0x6f76 bytes) 2024-04-23T21:22:05.030 INFO:teuthology.orchestra.run.smithi111.stdout:3576 read 0x4280923 thru 0x428df8f (0xd66d bytes) 2024-04-23T21:22:05.030 INFO:teuthology.orchestra.run.smithi111.stdout:3577 write 0x72fa68d thru 0x7308ddf (0xe753 bytes) 2024-04-23T21:22:05.034 INFO:teuthology.orchestra.run.smithi111.stdout:3578 read 0x10e4bc9 thru 0x10ec06e (0x74a6 bytes) 2024-04-23T21:22:05.034 INFO:teuthology.orchestra.run.smithi111.stdout:3579 read 0x3a7a39 thru 0x3ad44f (0x5a17 bytes) 2024-04-23T21:22:05.035 INFO:teuthology.orchestra.run.smithi111.stdout:3580 read 0x1bf6615 thru 0x1bf79d7 (0x13c3 bytes) 2024-04-23T21:22:05.042 INFO:teuthology.orchestra.run.smithi111.stdout:3583 writesame 0x9cc4d83 thru 0x9cd17c7 data_size 0x1(0xca45 bytes) 2024-04-23T21:22:05.043 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.045 INFO:teuthology.orchestra.run.smithi111.stdout:3587 trunc from 0x9cd17c8 to 0xbc8ab55 2024-04-23T21:22:05.050 INFO:teuthology.orchestra.run.smithi111.stdout:3588 writesame 0x8cf30b0 thru 0x8cf55e3 data_size 0x2(0x2534 bytes) 2024-04-23T21:22:05.056 INFO:teuthology.orchestra.run.smithi111.stdout:3590 trunc from 0xbc8ab55 to 0xb128d68 2024-04-23T21:22:05.062 INFO:teuthology.orchestra.run.smithi111.stdout:3591 read 0x30fe245 thru 0x310c2be (0xe07a bytes) 2024-04-23T21:22:05.062 INFO:teuthology.orchestra.run.smithi111.stdout:3593 read 0x7c5eb8b thru 0x7c647d7 (0x5c4d bytes) 2024-04-23T21:22:05.062 INFO:teuthology.orchestra.run.smithi111.stdout:3595 read 0x5034d93 thru 0x503a7ab (0x5a19 bytes) 2024-04-23T21:22:05.063 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.063 INFO:teuthology.orchestra.run.smithi111.stdout:3597 read 0x2491b20 thru 0x24a0bac (0xf08d bytes) 2024-04-23T21:22:05.063 INFO:teuthology.orchestra.run.smithi111.stdout:3598 writesame 0x7726983 thru 0x772904f data_size 0x3(0x26cd bytes) 2024-04-23T21:22:05.078 INFO:teuthology.orchestra.run.smithi111.stdout:3600 trunc from 0xb128d68 to 0x1da4f21 2024-04-23T21:22:05.111 INFO:teuthology.orchestra.run.smithi111.stdout:3601 write 0xe234ecb thru 0xe241492 (0xc5c8 bytes) 2024-04-23T21:22:05.116 INFO:teuthology.orchestra.run.smithi111.stdout:3602 writesame 0xd92de4d thru 0xd930ba2 data_size 0x7(0x2d56 bytes) 2024-04-23T21:22:05.119 INFO:teuthology.orchestra.run.smithi111.stdout:3605 read 0xb57d3d5 thru 0xb5868ce (0x94fa bytes) 2024-04-23T21:22:05.119 INFO:teuthology.orchestra.run.smithi111.stdout:3606 writesame 0x5c8866 thru 0x5d7424 data_size 0x1(0xebbf bytes) 2024-04-23T21:22:05.126 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:05.127+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:05.141 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:05.141 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:05.141 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.151 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:05.164 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.164 INFO:teuthology.orchestra.run.smithi111.stdout:3608 read 0x450a20c thru 0x45171ed (0xcfe2 bytes) 2024-04-23T21:22:05.164 INFO:teuthology.orchestra.run.smithi111.stdout:3610 writesame 0xa83ed73 thru 0xa84b5d5 data_size 0x1(0xc863 bytes) 2024-04-23T21:22:05.197 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:22:05.211 INFO:teuthology.orchestra.run.smithi111.stdout:3611 trunc from 0xe241493 to 0x101bf26 2024-04-23T21:22:05.225 INFO:teuthology.orchestra.run.smithi111.stdout:3615 read 0x4e1bf thru 0x58812 (0xa654 bytes) 2024-04-23T21:22:05.236 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:05.237+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:05.237 INFO:teuthology.orchestra.run.smithi111.stdout:3616 write 0x6ac8af3 thru 0x6acc2eb (0x37f9 bytes) 2024-04-23T21:22:05.240 INFO:teuthology.orchestra.run.smithi111.stdout:3617 punch from 0x6d7c50 to 0x6e390e, (0xbcbe bytes) 2024-04-23T21:22:05.241 INFO:teuthology.orchestra.run.smithi111.stdout:3618 writesame 0x6a89c11 thru 0x6a92b8b data_size 0x1(0x8f7b bytes) 2024-04-23T21:22:05.273 INFO:teuthology.orchestra.run.smithi111.stdout:3620 write 0xe04bdaa thru 0xe057ac6 (0xbd1d bytes) 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:3621 read 0x29b57c1 thru 0x29b96d8 (0x3f18 bytes) 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:3622 read 0x3472f0d thru 0x34803bf (0xd4b3 bytes) 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:3625 read 0x2bb10d3 thru 0x2bb4088 (0x2fb6 bytes) 2024-04-23T21:22:05.276 INFO:teuthology.orchestra.run.smithi111.stdout:3626 write 0x696f5b thru 0x69ef8a (0x8030 bytes) 2024-04-23T21:22:05.278 INFO:teuthology.orchestra.run.smithi111.stdout:3627 read 0x4461df5 thru 0x446f122 (0xd32e bytes) 2024-04-23T21:22:05.278 INFO:teuthology.orchestra.run.smithi111.stdout:3628 write 0x86929fb thru 0x86a0414 (0xda1a bytes) 2024-04-23T21:22:05.281 INFO:teuthology.orchestra.run.smithi111.stdout:3632 punch from 0x1897ee2 to 0x18a5a5a, (0xdb78 bytes) 2024-04-23T21:22:05.281 INFO:teuthology.orchestra.run.smithi111.stdout:3633 write 0x34afdf2 thru 0x34b12f2 (0x1501 bytes) 2024-04-23T21:22:05.284 INFO:teuthology.orchestra.run.smithi111.stdout:3634 read 0xca13aaa thru 0xca1cf0d (0x9464 bytes) 2024-04-23T21:22:05.284 INFO:teuthology.orchestra.run.smithi111.stdout:3635 writesame 0x4b8372c thru 0x4b8c966 data_size 0x1(0x923b bytes) 2024-04-23T21:22:05.301 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.301 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:05.301 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:05.301 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:05.301 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.302 INFO:teuthology.orchestra.run.smithi111.stdout:3636 read 0xa5bcf00 thru 0xa5c4602 (0x7703 bytes) 2024-04-23T21:22:05.302 INFO:teuthology.orchestra.run.smithi111.stdout:3637 read 0x4ed4726 thru 0x4edc18a (0x7a65 bytes) 2024-04-23T21:22:05.302 INFO:teuthology.orchestra.run.smithi111.stdout:3640 read 0xff6b5 thru 0x10f2a1 (0xfbed bytes) 2024-04-23T21:22:05.302 INFO:teuthology.orchestra.run.smithi111.stdout:3641 write 0x9ba0024 thru 0x9ba6064 (0x6041 bytes) 2024-04-23T21:22:05.306 INFO:teuthology.orchestra.run.smithi111.stdout:3642 read 0x38e9470 thru 0x38f69ae (0xd53f bytes) 2024-04-23T21:22:05.306 INFO:teuthology.orchestra.run.smithi111.stdout:3643 write 0xb877c97 thru 0xb881a97 (0x9e01 bytes) 2024-04-23T21:22:05.310 INFO:teuthology.orchestra.run.smithi111.stdout:3644 read 0xb795860 thru 0xb7a2d2e (0xd4cf bytes) 2024-04-23T21:22:05.310 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.310 INFO:teuthology.orchestra.run.smithi111.stdout:3646 punch from 0x325c907 to 0x325fdd9, (0x34d2 bytes) 2024-04-23T21:22:05.310 INFO:teuthology.orchestra.run.smithi111.stdout:3648 writesame 0xd3ad5ae thru 0xd3adf92 data_size 0x1(0x9e5 bytes) 2024-04-23T21:22:05.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:05.314 INFO:teuthology.orchestra.run.smithi111.stdout:3649 read 0x4098cc7 thru 0x409af29 (0x2263 bytes) 2024-04-23T21:22:05.314 INFO:teuthology.orchestra.run.smithi111.stdout:3650 write 0xc060a6 thru 0xc0cda0 (0x6cfb bytes) 2024-04-23T21:22:05.325 INFO:teuthology.orchestra.run.smithi111.stdout:3651 punch from 0x93a08f8 to 0x93a2195, (0x189d bytes) 2024-04-23T21:22:05.325 INFO:teuthology.orchestra.run.smithi111.stdout:3652 read 0x8a0fe90 thru 0x8a15e7d (0x5fee bytes) 2024-04-23T21:22:05.326 INFO:teuthology.orchestra.run.smithi111.stdout:3653 writesame 0x30a5302 thru 0x30b3417 data_size 0x2f(0xe116 bytes) 2024-04-23T21:22:05.328 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:05.329+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:05.329 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.329 INFO:teuthology.orchestra.run.smithi111.stdout:3656 trunc from 0xe057ac7 to 0xde9a0a5 2024-04-23T21:22:05.338 INFO:teuthology.orchestra.run.smithi111.stdout:3657 punch from 0x2d87b4b to 0x2d8b91c, (0x3dd1 bytes) 2024-04-23T21:22:05.339 INFO:teuthology.orchestra.run.smithi111.stdout:3658 punch from 0x944c309 to 0x945172a, (0x5421 bytes) 2024-04-23T21:22:05.339 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.339 INFO:teuthology.orchestra.run.smithi111.stdout:3662 read 0x7d5a78c thru 0x7d6a5a7 (0xfe1c bytes) 2024-04-23T21:22:05.342 INFO:teuthology.orchestra.run.smithi111.stdout:3663 trunc from 0xde9a0a5 to 0xa7adbf7 2024-04-23T21:22:05.353 INFO:teuthology.orchestra.run.smithi111.stdout:3664 punch from 0x338af88 to 0x338b42f, (0x4a7 bytes) 2024-04-23T21:22:05.353 INFO:teuthology.orchestra.run.smithi111.stdout:3666 write 0xce256d thru 0xcea4f3 (0x7f87 bytes) 2024-04-23T21:22:05.355 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.355 INFO:teuthology.orchestra.run.smithi111.stdout:3668 read 0x3378d8 thru 0x34265a (0xad83 bytes) 2024-04-23T21:22:05.356 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.356 INFO:teuthology.orchestra.run.smithi111.stdout:3670 write 0x396e862 thru 0x3979a03 (0xb1a2 bytes) 2024-04-23T21:22:05.368 INFO:teuthology.orchestra.run.smithi111.stdout:3671 trunc from 0xa7adbf7 to 0x3221c3b 2024-04-23T21:22:05.380 INFO:teuthology.orchestra.run.smithi111.stdout:3673 read 0x17af285 thru 0x17b5388 (0x6104 bytes) 2024-04-23T21:22:05.382 INFO:teuthology.orchestra.run.smithi111.stdout:3674 trunc from 0x3221c3b to 0x1155a9b 2024-04-23T21:22:05.398 INFO:teuthology.orchestra.run.smithi111.stdout:3675 trunc from 0x1155a9b to 0x3e2a858 2024-04-23T21:22:05.404 INFO:teuthology.orchestra.run.smithi111.stdout:3676 read 0x1fb1df7 thru 0x1fb5093 (0x329d bytes) 2024-04-23T21:22:05.404 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.404 INFO:teuthology.orchestra.run.smithi111.stdout:3679 trunc from 0x3e2a858 to 0x3dad5ea 2024-04-23T21:22:05.410 INFO:teuthology.orchestra.run.smithi111.stdout:3680 read 0x1eb9850 thru 0x1ebfdd7 (0x6588 bytes) 2024-04-23T21:22:05.410 INFO:teuthology.orchestra.run.smithi111.stdout:3681 read 0x38460fb thru 0x384b1c6 (0x50cc bytes) 2024-04-23T21:22:05.410 INFO:teuthology.orchestra.run.smithi111.stdout:3685 punch from 0x1a97163 to 0x1a9b22e, (0x40cb bytes) 2024-04-23T21:22:05.410 INFO:teuthology.orchestra.run.smithi111.stdout:3687 read 0x7a80df thru 0x7ad024 (0x4f46 bytes) 2024-04-23T21:22:05.410 INFO:teuthology.orchestra.run.smithi111.stdout:3689 read 0x21e97b thru 0x2241c5 (0x584b bytes) 2024-04-23T21:22:05.411 INFO:teuthology.orchestra.run.smithi111.stdout:3691 write 0x2a52c00 thru 0x2a5c1e0 (0x95e1 bytes) 2024-04-23T21:22:05.414 INFO:teuthology.orchestra.run.smithi111.stdout:3694 read 0x3a6d5cc thru 0x3a77a9c (0xa4d1 bytes) 2024-04-23T21:22:05.414 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.414 INFO:teuthology.orchestra.run.smithi111.stdout:3696 writesame 0x423eb7 thru 0x428af2 data_size 0x22(0x4c3c bytes) 2024-04-23T21:22:05.416 INFO:teuthology.orchestra.run.smithi111.stdout:3697 read 0x38a18a4 thru 0x38a8ccb (0x7428 bytes) 2024-04-23T21:22:05.416 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.416 INFO:teuthology.orchestra.run.smithi111.stdout:3700 read 0x23613d2 thru 0x236c1ab (0xadda bytes) 2024-04-23T21:22:05.421 INFO:teuthology.orchestra.run.smithi111.stdout:3702 writesame 0x4218509 thru 0x42199f5 data_size 0x1(0x14ed bytes) 2024-04-23T21:22:05.421 INFO:teuthology.orchestra.run.smithi111.stdout:3703 punch from 0x3333c7a to 0x3338986, (0x4d0c bytes) 2024-04-23T21:22:05.429 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:05.430+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:05.430 INFO:teuthology.orchestra.run.smithi111.stdout:3704 write 0x78107d5 thru 0x7810f75 (0x7a1 bytes) 2024-04-23T21:22:05.433 INFO:teuthology.orchestra.run.smithi111.stdout:3705 read 0x64644dc thru 0x6465d7a (0x189f bytes) 2024-04-23T21:22:05.433 INFO:teuthology.orchestra.run.smithi111.stdout:3709 read 0x1e682fd thru 0x1e69418 (0x111c bytes) 2024-04-23T21:22:05.446 INFO:teuthology.orchestra.run.smithi111.stdout:3711 write 0xd281e6c thru 0xd28bae5 (0x9c7a bytes) 2024-04-23T21:22:05.450 INFO:teuthology.orchestra.run.smithi111.stdout:3712 punch from 0xb992435 to 0xb99b4c0, (0x908b bytes) 2024-04-23T21:22:05.450 INFO:teuthology.orchestra.run.smithi111.stdout:3713 read 0xbde6af8 thru 0xbdf329d (0xc7a6 bytes) 2024-04-23T21:22:05.451 INFO:teuthology.orchestra.run.smithi111.stdout:3715 read 0x54201d6 thru 0x542ad20 (0xab4b bytes) 2024-04-23T21:22:05.451 INFO:teuthology.orchestra.run.smithi111.stdout:3718 write 0x135ac2b thru 0x135eddc (0x41b2 bytes) 2024-04-23T21:22:05.451 INFO:teuthology.orchestra.run.smithi111.stdout:3719 read 0xb99feb9 thru 0xb9aef19 (0xf061 bytes) 2024-04-23T21:22:05.458 INFO:teuthology.orchestra.run.smithi111.stdout:3720 write 0xdb3a961 thru 0xdb47afa (0xd19a bytes) 2024-04-23T21:22:05.462 INFO:teuthology.orchestra.run.smithi111.stdout:3723 punch from 0x27c3084 to 0x27c7767, (0x46e3 bytes) 2024-04-23T21:22:05.462 INFO:teuthology.orchestra.run.smithi111.stdout:3724 write 0x2658fc7 thru 0x26649af (0xb9e9 bytes) 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.812082263000001, 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.467 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:05.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:05.471 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.812061139000001, 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.475 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.812031872, 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.476 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:05.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.480 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.81201111, 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:05.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811992084, 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:05.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.489 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811973245000001, 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:05.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:05.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:05.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811939842999999, 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:05.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811920097, 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.499 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:05.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:05.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:05.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811901993999999, 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811882793000001, 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.507 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:05.511 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811857394, 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:05.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.515 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811832269, 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.520 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811807258, 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.521 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.522 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:05.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:05.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811783689, 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:05.530 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:05.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.534 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811760533999999, 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.535 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:05.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:05.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811739791000001, 2024-04-23T21:22:05.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.540 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:05.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:05.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:05.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:05.546 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.81172149, 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.547 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:05.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.553 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.81170391, 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.554 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:05.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:05.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:05.557 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 13.811674387, 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:05.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.0724193230000001, 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.562 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:05.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:05.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:05.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.568 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3725 write 0x59dd4a thru 0x5a789c (0x9b53 bytes) 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3728 writesame 0xaca30ce thru 0xacabe8e data_size 0x1(0x8dc1 bytes) 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3729 write 0x23656ea thru 0x236fa07 (0xa31e bytes) 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3733 write 0x89f5d23 thru 0x89fdba9 (0x7e87 bytes) 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3734 punch from 0x17be14a to 0x17c5acb, (0x7981 bytes) 2024-04-23T21:22:05.571 INFO:teuthology.orchestra.run.smithi111.stdout:3735 trunc from 0xdb47afb to 0x75d2854 2024-04-23T21:22:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:3739 trunc from 0x75d2854 to 0x18e756b 2024-04-23T21:22:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:3740 write 0x301f1ed thru 0x302bfde (0xcdf2 bytes) 2024-04-23T21:22:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:3741 writesame 0x79b35e9 thru 0x79bbb57 data_size 0x1(0x856f bytes) 2024-04-23T21:22:05.572 INFO:teuthology.orchestra.run.smithi111.stdout:3742 writesame 0x30cca91 thru 0x30d3c4f data_size 0x1(0x71bf bytes) 2024-04-23T21:22:05.572 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:05.570+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:05.582 INFO:teuthology.orchestra.run.smithi111.stdout:3743 writesame 0xea42b1f thru 0xea4308a data_size 0x2(0x56c bytes) 2024-04-23T21:22:05.582 INFO:teuthology.orchestra.run.smithi111.stdout:3744 write 0x43b50cb thru 0x43c2b68 (0xda9e bytes) 2024-04-23T21:22:05.583 INFO:teuthology.orchestra.run.smithi111.stdout:3745 writesame 0x4d11f1d thru 0x4d12eeb data_size 0x13(0xfcf bytes) 2024-04-23T21:22:05.585 INFO:teuthology.orchestra.run.smithi111.stdout:3746 punch from 0xb3562df to 0xb364e2f, (0xeb50 bytes) 2024-04-23T21:22:05.585 INFO:teuthology.orchestra.run.smithi111.stdout:3747 read 0x59522 thru 0x64c82 (0xb761 bytes) 2024-04-23T21:22:05.586 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.586 INFO:teuthology.orchestra.run.smithi111.stdout:3750 write 0x1dcecb6 thru 0x1dd0ad3 (0x1e1e bytes) 2024-04-23T21:22:05.590 INFO:teuthology.orchestra.run.smithi111.stdout:3752 punch from 0x615bb20 to 0x615eef2, (0x33d2 bytes) 2024-04-23T21:22:05.590 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.590 INFO:teuthology.orchestra.run.smithi111.stdout:3754 write 0x8623d5 thru 0x86619e (0x3dca bytes) 2024-04-23T21:22:05.591 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.591 INFO:teuthology.orchestra.run.smithi111.stdout:3758 write 0xb2affc5 thru 0xb2b5168 (0x51a4 bytes) 2024-04-23T21:22:05.596 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.597 INFO:teuthology.orchestra.run.smithi111.stdout:3761 read 0xfbbddb thru 0xfc7981 (0xbba7 bytes) 2024-04-23T21:22:05.597 INFO:teuthology.orchestra.run.smithi111.stdout:3764 read 0xdafa082 thru 0xdafbecf (0x1e4e bytes) 2024-04-23T21:22:05.601 INFO:teuthology.orchestra.run.smithi111.stdout:3765 trunc from 0xea4308b to 0xb6f4798 2024-04-23T21:22:05.606 INFO:teuthology.orchestra.run.smithi111.stdout:3766 write 0x8669634 thru 0x866cbc5 (0x3592 bytes) 2024-04-23T21:22:05.609 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.609 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.615 INFO:teuthology.orchestra.run.smithi111.stdout:3770 writesame 0xc0f5031 thru 0xc103903 data_size 0x1(0xe8d3 bytes) 2024-04-23T21:22:05.616 INFO:teuthology.orchestra.run.smithi111.stdout:3771 punch from 0xbf7eb3d to 0xbf8189c, (0x2d5f bytes) 2024-04-23T21:22:05.616 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.616 INFO:teuthology.orchestra.run.smithi111.stdout:3773 read 0xa210895 thru 0xa21ad07 (0xa473 bytes) 2024-04-23T21:22:05.616 INFO:teuthology.orchestra.run.smithi111.stdout:3774 write 0x513360b thru 0x5136e96 (0x388c bytes) 2024-04-23T21:22:05.620 INFO:teuthology.orchestra.run.smithi111.stdout:3775 read 0x4e9bfc6 thru 0x4eaaaab (0xeae6 bytes) 2024-04-23T21:22:05.621 INFO:teuthology.orchestra.run.smithi111.stdout:3776 read 0x57f1e7b thru 0x57fb88a (0x9a10 bytes) 2024-04-23T21:22:05.632 INFO:teuthology.orchestra.run.smithi111.stdout:3777 trunc from 0xc103904 to 0x1fbd818 2024-04-23T21:22:05.658 INFO:teuthology.orchestra.run.smithi111.stdout:3778 write 0x66a6647 thru 0x66acd0c (0x66c6 bytes) 2024-04-23T21:22:05.664 INFO:teuthology.orchestra.run.smithi111.stdout:3779 trunc from 0x66acd0d to 0x88c9860 2024-04-23T21:22:05.670 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:05.671+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:05.671 INFO:teuthology.orchestra.run.smithi111.stdout:3780 read 0x39ab550 thru 0x39b1e65 (0x6916 bytes) 2024-04-23T21:22:05.677 INFO:teuthology.orchestra.run.smithi111.stdout:3783 trunc from 0x88c9860 to 0x31b75a7 2024-04-23T21:22:05.683 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.683 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:05.683 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:05.683 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.688 INFO:teuthology.orchestra.run.smithi111.stdout:3785 read 0xc779c6 thru 0xc7c57a (0x4bb5 bytes) 2024-04-23T21:22:05.689 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:05.689 INFO:teuthology.orchestra.run.smithi111.stdout:3787 write 0x151563c thru 0x15166b1 (0x1076 bytes) 2024-04-23T21:22:05.690 INFO:teuthology.orchestra.run.smithi111.stdout:3788 read 0x16eaa3c thru 0x16ee404 (0x39c9 bytes) 2024-04-23T21:22:05.690 INFO:teuthology.orchestra.run.smithi111.stdout:3789 read 0x11da370 thru 0x11e313a (0x8dcb bytes) 2024-04-23T21:22:05.690 INFO:teuthology.orchestra.run.smithi111.stdout:3790 read 0x185775 thru 0x191e03 (0xc68f bytes) 2024-04-23T21:22:05.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:05.700 INFO:teuthology.orchestra.run.smithi111.stdout:3791 writesame 0x5615bf7 thru 0x5619cb2 data_size 0x6(0x40bc bytes) 2024-04-23T21:22:05.713 INFO:teuthology.orchestra.run.smithi111.stdout:3792 write 0xaefb201 thru 0xaefc209 (0x1009 bytes) 2024-04-23T21:22:05.716 INFO:teuthology.orchestra.run.smithi111.stdout:3793 write 0x5378960 thru 0x537995d (0xffe bytes) 2024-04-23T21:22:05.720 INFO:teuthology.orchestra.run.smithi111.stdout:3794 punch from 0x7acdae8 to 0x7ad44be, (0x69d6 bytes) 2024-04-23T21:22:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:3796 trunc from 0xaefc20a to 0x1b9d789 2024-04-23T21:22:05.743 INFO:teuthology.orchestra.run.smithi111.stdout:3797 punch from 0x1ab2e6c to 0x1ab7a26, (0x4bba bytes) 2024-04-23T21:22:05.743 INFO:teuthology.orchestra.run.smithi111.stdout:3799 punch from 0x24d5d0 to 0x2505fc, (0x302c bytes) 2024-04-23T21:22:05.751 INFO:teuthology.orchestra.run.smithi111.stdout:3801 write 0x2cfa9b8 thru 0x2d08bf3 (0xe23c bytes) 2024-04-23T21:22:05.766 INFO:teuthology.orchestra.run.smithi111.stdout:3802 write 0x5c7a3da thru 0x5c7ecb6 (0x48dd bytes) 2024-04-23T21:22:05.771 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:05.772+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:05.782 INFO:teuthology.orchestra.run.smithi111.stdout:3804 write 0xa9be4cb thru 0xa9c860d (0xa143 bytes) 2024-04-23T21:22:05.786 INFO:teuthology.orchestra.run.smithi111.stdout:3806 read 0x7100e7a thru 0x710f927 (0xeaae bytes) 2024-04-23T21:22:05.786 INFO:teuthology.orchestra.run.smithi111.stdout:3807 write 0x223954c thru 0x223efb0 (0x5a65 bytes) 2024-04-23T21:22:05.797 INFO:teuthology.orchestra.run.smithi111.stdout:3808 write 0xd319418 thru 0xd322386 (0x8f6f bytes) 2024-04-23T21:22:05.801 INFO:teuthology.orchestra.run.smithi111.stdout:3811 write 0x25622c3 thru 0x2563176 (0xeb4 bytes) 2024-04-23T21:22:05.803 INFO:teuthology.orchestra.run.smithi111.stdout:3812 read 0xb2dfc1b thru 0xb2eeeb9 (0xf29f bytes) 2024-04-23T21:22:05.804 INFO:teuthology.orchestra.run.smithi111.stdout:3814 write 0x97f8dc1 thru 0x98073fe (0xe63e bytes) 2024-04-23T21:22:05.806 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.806 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:05.806 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:05.806 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:05.806 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:05.810 INFO:teuthology.orchestra.run.smithi111.stdout:3815 read 0x13a09f thru 0x1464bb (0xc41d bytes) 2024-04-23T21:22:05.811 INFO:teuthology.orchestra.run.smithi111.stdout:3817 writesame 0x60d8dcb thru 0x60dabf2 data_size 0x14(0x1e28 bytes) 2024-04-23T21:22:05.813 INFO:teuthology.orchestra.run.smithi111.stdout:3818 writesame 0x97c7bd2 thru 0x97cf256 data_size 0x1(0x7685 bytes) 2024-04-23T21:22:05.816 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:05.828 INFO:teuthology.orchestra.run.smithi111.stdout:3819 write 0xa86838e thru 0xa876d1d (0xe990 bytes) 2024-04-23T21:22:05.832 INFO:teuthology.orchestra.run.smithi111.stdout:3820 read 0x58817ea thru 0x5884cce (0x34e5 bytes) 2024-04-23T21:22:05.832 INFO:teuthology.orchestra.run.smithi111.stdout:3822 writesame 0x2935252 thru 0x293c837 data_size 0x1(0x75e6 bytes) 2024-04-23T21:22:05.846 INFO:teuthology.orchestra.run.smithi111.stdout:3823 clone 30 order 23 su 1048576 sc 3 2024-04-23T21:22:05.871 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:05.872+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:05.920 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499913331000002, 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499885968000001, 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499871718999998, 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499849476000001, 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:22:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:22:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499783989999997, 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499758247999999, 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:22:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499734238999999, 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499711679999997, 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499691876999997, 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:22:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:22:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499672390999997, 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:22:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499652591999997, 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499632890000001, 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:22:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499576763999997, 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499539542999997, 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499489230000002, 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:22:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:22:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499456334999998, 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:22:05.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.988 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499434031, 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.989 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:05.995 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.499413615000002, 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:22:05.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:22:05.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:05.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61148 3.13 3:cc4c0078:::rbd_data.1118e08f6c14.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.527673933, 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033335561999999999, 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61148 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392816+0000", 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0609999999999998e-06 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407607+0000", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014790792000000001 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407632+0000", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5092999999999998e-05 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407646+0000", 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4134999999999999e-05 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407719+0000", 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3206000000000003e-05 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407831+0000", 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111797 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:06.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408130+0000", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029870400000000001 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424207+0000", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016077281999999998 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426036+0000", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001829378 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:06.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426098+0000", 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1693999999999996e-05 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426149+0000", 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1419999999999999e-05 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.79304310099999997, 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:06.005 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:06.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:06.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:06.010 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:06.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:06.011 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:06.011 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:06.012 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:06.013 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:06.013+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:06.113 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:06.115+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:06.115 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:06.115 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:06.115 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:06.115 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:06.124 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:06.214 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:06.215+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:06.274 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:06.275 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:06.275 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:06.275 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:06.275 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:06.277 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone29 intact 2024-04-23T21:22:06.285 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:06.314 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:06.315+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:06.396 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 17.455640889000001, 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:06.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:06.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53205 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 2932736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.611012329999999, 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013002390000000001, 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53205 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785414+0000", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7599999999999997e-07 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785417+0000", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0019999999999998e-06 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785427+0000", 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5929999999999995e-06 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785441+0000", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4764e-05 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798225+0000", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012784007 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798310+0000", 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4618999999999997e-05 2024-04-23T21:22:06.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798416+0000", 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010582900000000001 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53206 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 3309568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610975729, 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013168311, 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.406 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53206 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785451+0000", 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785453+0000", 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785460+0000", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.742e-06 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785464+0000", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5390000000000001e-06 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798454+0000", 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012989908999999999 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798540+0000", 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6206999999999996e-05 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798618+0000", 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7956999999999999e-05 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.410 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53258 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 753664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610712564, 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013389243, 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53258 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785714+0000", 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6400000000000001e-07 2024-04-23T21:22:06.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785724+0000", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8560000000000002e-06 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785736+0000", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2339e-05 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785744+0000", 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8050000000000007e-06 2024-04-23T21:22:06.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798657+0000", 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012913345999999999 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798736+0000", 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8627000000000005e-05 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799102+0000", 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000366406 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53259 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610669829000001, 2024-04-23T21:22:06.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013579291, 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53259 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:06.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785757+0000", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785759+0000", 2024-04-23T21:22:06.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.481e-06 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785769+0000", 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0234000000000001e-05 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785775+0000", 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.648e-06 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799145+0000", 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013370253 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799229+0000", 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3820999999999996e-05 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799335+0000", 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000106226 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53260 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610642664, 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01398165, 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53260 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785784+0000", 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.27e-07 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785786+0000", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2520000000000002e-06 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785792+0000", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6510000000000004e-06 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785798+0000", 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3170000000000001e-06 2024-04-23T21:22:06.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799366+0000", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013568573 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799453+0000", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6549000000000001e-05 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799765+0000", 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031188100000000002 2024-04-23T21:22:06.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53261 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610621347, 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014180051000000001, 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53261 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785805+0000", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785807+0000", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0190000000000001e-06 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785814+0000", 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:06.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785820+0000", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1279999999999996e-06 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799793+0000", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013973167 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799884+0000", 2024-04-23T21:22:06.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1049000000000002e-05 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799984+0000", 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010002500000000001 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53262 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1671168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.61059813, 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014362929, 2024-04-23T21:22:06.428 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53262 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785830+0000", 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.131e-06 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785837+0000", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.849e-06 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785842+0000", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3859999999999998e-06 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800015+0000", 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014173008000000001 2024-04-23T21:22:06.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800098+0000", 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2268999999999994e-05 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800191+0000", 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2818000000000005e-05 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53263 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1851392~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610576307000001, 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014770998, 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53263 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785850+0000", 2024-04-23T21:22:06.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6199999999999998e-07 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785852+0000", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.046e-06 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785858+0000", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4799999999999998e-06 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785864+0000", 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1279999999999999e-06 2024-04-23T21:22:06.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800236+0000", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014372433 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800327+0000", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0816000000000006e-05 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800620+0000", 2024-04-23T21:22:06.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00029363300000000001 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53264 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610555879, 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014801979999999999, 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.437 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53264 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785873+0000", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9160000000000001e-06 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785880+0000", 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6599999999999998e-06 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785885+0000", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.113e-06 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800660+0000", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775352 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800672+0000", 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1528e-05 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.440 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53265 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610523787, 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014804595, 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.441 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53265 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785905+0000", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4250000000000001e-06 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785912+0000", 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3889999999999999e-06 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785924+0000", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1953999999999999e-05 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800700+0000", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775307 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800707+0000", 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0489999999999998e-06 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.444 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53266 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610474910000001, 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014787503, 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53266 2024-04-23T21:22:06.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9999999999999997e-07 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785954+0000", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.649e-06 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785969+0000", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5058999999999999e-05 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785975+0000", 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9529999999999996e-06 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800730+0000", 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014755111 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800738+0000", 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1310000000000006e-06 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53267 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610441948, 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014785247, 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53267 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2e-07 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785992+0000", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8059999999999995e-06 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785999+0000", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3520000000000001e-06 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786006+0000", 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5549999999999999e-06 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800760+0000", 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014754259 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800769+0000", 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.755e-06 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53268 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.452 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610405700999999, 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014799623, 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53268 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:06.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786021+0000", 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786023+0000", 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5569999999999998e-06 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786031+0000", 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.644e-06 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786036+0000", 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4890000000000003e-06 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800808+0000", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014771467 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800820+0000", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1924e-05 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53269 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:06.456 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.610375962999999, 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014798240000000001, 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53269 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:06.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786057+0000", 2024-04-23T21:22:06.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3040000000000001e-06 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786065+0000", 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8640000000000002e-06 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786077+0000", 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1595e-05 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800840+0000", 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014762542999999999 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800848+0000", 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4249999999999998e-06 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:61142 3.4 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:06.460 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 6.0182722330000002, 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.019170651, 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 61142 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:06.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378156+0000", 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2989999999999999e-06 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383079+0000", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049228930000000002 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383111+0000", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2338000000000001e-05 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383125+0000", 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4039999999999999e-05 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383226+0000", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010133599999999999 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383338+0000", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000111777 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384242+0000", 2024-04-23T21:22:06.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00090330900000000003 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.395145+0000", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010903042999999999 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397245+0000", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002099952 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397289+0000", 2024-04-23T21:22:06.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4792000000000001e-05 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397324+0000", 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4872000000000001e-05 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63700 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:06.466 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.2272354000000001, 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027843201000000001, 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63700 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169192+0000", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5069999999999999e-06 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169200+0000", 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1429999999999998e-06 2024-04-23T21:22:06.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169214+0000", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4422e-05 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169221+0000", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6699999999999997e-06 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169280+0000", 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.906e-05 2024-04-23T21:22:06.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169343+0000", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2478999999999996e-05 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.174410+0000", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005067078 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.195126+0000", 2024-04-23T21:22:06.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020716472 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196931+0000", 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018052929999999999 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196984+0000", 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2627999999999998e-05 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.197034+0000", 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9449e-05 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63757 3.19 3:9ccb22ce:::rbd_data.1118139b0f86.0000000000000025:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 1.1089094959999999, 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.015960412, 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.472 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63757 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287517+0000", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1900000000000001e-07 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287522+0000", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287533+0000", 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.079e-05 2024-04-23T21:22:06.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287540+0000", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2019999999999998e-06 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287617+0000", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7014000000000005e-05 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287667+0000", 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9421e-05 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 3", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.290130+0000", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0024636150000000002 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303396+0000", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013265476 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303410+0000", 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3946e-05 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303446+0000", 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.612e-05 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303477+0000", 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0839000000000001e-05 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63940 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:06.477 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.92380894000000002, 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.016049744000000001, 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63940 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472618+0000", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.61e-07 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472622+0000", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1690000000000002e-06 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472632+0000", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0399e-05 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472637+0000", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2310000000000004e-06 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472674+0000", 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6465999999999999e-05 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472718+0000", 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580999999999999e-05 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.475549+0000", 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028307340000000001 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488593+0000", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013043406 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488604+0000", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1649e-05 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488636+0000", 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1622999999999999e-05 2024-04-23T21:22:06.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488667+0000", 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0725000000000001e-05 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:64217 3.6 3:67a8ca51:::rbd_data.1118139b0f86.0000000000000049:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.57987038099999999, 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013303083, 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:06.483 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 64217 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816556+0000", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2500000000000005e-07 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816560+0000", 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2030000000000002e-06 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816571+0000", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0514e-05 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816576+0000", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9169999999999997e-06 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816618+0000", 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2190000000000001e-05 2024-04-23T21:22:06.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816710+0000", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2008000000000004e-05 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.818915+0000", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002205046 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829742+0000", 2024-04-23T21:22:06.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010826987 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829794+0000", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2126000000000002e-05 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829828+0000", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4406999999999998e-05 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829858+0000", 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9960000000000001e-05 2024-04-23T21:22:06.488 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.489 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:06.489 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:06.489 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:06.490 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-04-23T21:22:06.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:06.491 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:06.491+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:06.591 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:06.592+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:06.642 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:06.642 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:06.642 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:06.642 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:06.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:06.691 INFO:teuthology.orchestra.run.smithi111.stdout:3824 read 0x9083b83 thru 0x908d4bd (0x993b bytes) 2024-04-23T21:22:06.692 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:06.693+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:06.693 INFO:teuthology.orchestra.run.smithi111.stdout:3825 write 0x9ac3983 thru 0x9ac79ce (0x404c bytes) 2024-04-23T21:22:06.700 INFO:teuthology.orchestra.run.smithi111.stdout:3828 writesame 0x3ea8c24 thru 0x3eb7861 data_size 0xb(0xec3e bytes) 2024-04-23T21:22:06.720 INFO:teuthology.orchestra.run.smithi111.stdout:3829 trunc from 0xd322387 to 0xaf6179 2024-04-23T21:22:06.728 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:06.781 INFO:teuthology.orchestra.run.smithi111.stdout:3830 write 0xbe04e91 thru 0xbe116ae (0xc81e bytes) 2024-04-23T21:22:06.792 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:06.794+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:06.793 INFO:teuthology.orchestra.run.smithi111.stdout:3832 write 0xd749782 thru 0xd756e92 (0xd711 bytes) 2024-04-23T21:22:06.804 INFO:teuthology.orchestra.run.smithi111.stdout:3834 write 0xed06d18 thru 0xed0ea66 (0x7d4f bytes) 2024-04-23T21:22:06.806 INFO:teuthology.orchestra.run.smithi111.stdout:3835 read 0xe81db32 thru 0xe826ef5 (0x93c4 bytes) 2024-04-23T21:22:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:3836 read 0xad37144 thru 0xad42faa (0xbe67 bytes) 2024-04-23T21:22:06.810 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:06.810 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:06.810 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:06.810 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:06.811 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:06.812 INFO:teuthology.orchestra.run.smithi111.stdout:3837 trunc from 0xed0ea67 to 0xa48baa6 2024-04-23T21:22:06.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:06.823 INFO:teuthology.orchestra.run.smithi111.stdout:3838 read 0x9cc3e22 thru 0x9cc4cca (0xea9 bytes) 2024-04-23T21:22:06.823 INFO:teuthology.orchestra.run.smithi111.stdout:3840 writesame 0x6996483 thru 0x699d577 data_size 0x77(0x70f5 bytes) 2024-04-23T21:22:06.824 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.824 INFO:teuthology.orchestra.run.smithi111.stdout:3842 punch from 0x5c0e781 to 0x5c11cbb, (0x353a bytes) 2024-04-23T21:22:06.824 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.834 INFO:teuthology.orchestra.run.smithi111.stdout:3844 trunc from 0xa48baa6 to 0x12f78fd 2024-04-23T21:22:06.847 INFO:teuthology.orchestra.run.smithi111.stdout:3845 read 0x84e393 thru 0x84ff81 (0x1bef bytes) 2024-04-23T21:22:06.848 INFO:teuthology.orchestra.run.smithi111.stdout:3847 punch from 0x12dc6cd to 0x12e3273, (0x6ba6 bytes) 2024-04-23T21:22:06.848 INFO:teuthology.orchestra.run.smithi111.stdout:3850 punch from 0x6db1ea to 0x6dcc92, (0x1aa8 bytes) 2024-04-23T21:22:06.856 INFO:teuthology.orchestra.run.smithi111.stdout:3853 trunc from 0x12f78fd to 0x814c5af 2024-04-23T21:22:06.864 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:06.865 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:06.869 INFO:teuthology.orchestra.run.smithi111.stdout:3854 writesame 0x96c443f thru 0x96c7201 data_size 0x37(0x2dc3 bytes) 2024-04-23T21:22:06.877 INFO:teuthology.orchestra.run.smithi111.stdout:3855 write 0xa38a59c thru 0xa395369 (0xadce bytes) 2024-04-23T21:22:06.880 INFO:teuthology.orchestra.run.smithi111.stdout:3856 read 0x803c108 thru 0x803d6be (0x15b7 bytes) 2024-04-23T21:22:06.881 INFO:teuthology.orchestra.run.smithi111.stdout:3857 trunc from 0xa39536a to 0x9b5b5bc 2024-04-23T21:22:06.884 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:06.885 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:06.889 INFO:teuthology.orchestra.run.smithi111.stdout:3858 read 0x44b0a84 thru 0x44b60e3 (0x5660 bytes) 2024-04-23T21:22:06.889 INFO:teuthology.orchestra.run.smithi111.stdout:3859 write 0x5f398ee thru 0x5f43e43 (0xa556 bytes) 2024-04-23T21:22:06.893 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:06.894+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:06.893 INFO:teuthology.orchestra.run.smithi111.stdout:3861 punch from 0x8e7ebab to 0x8e81900, (0x2d55 bytes) 2024-04-23T21:22:06.894 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.894 INFO:teuthology.orchestra.run.smithi111.stdout:3864 read 0x5c0d36e thru 0x5c10e3e (0x3ad1 bytes) 2024-04-23T21:22:06.894 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.894 INFO:teuthology.orchestra.run.smithi111.stdout:3867 write 0x830642b thru 0x831501a (0xebf0 bytes) 2024-04-23T21:22:06.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:06.903 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:06.905 INFO:teuthology.orchestra.run.smithi111.stdout:3869 writesame 0xb26b4f5 thru 0xb274515 data_size 0x31(0x9021 bytes) 2024-04-23T21:22:06.905 INFO:teuthology.orchestra.run.smithi111.stdout:3870 read 0x9b0b764 thru 0x9b1a0e1 (0xe97e bytes) 2024-04-23T21:22:06.905 INFO:teuthology.orchestra.run.smithi111.stdout:3873 trunc from 0xb274516 to 0xaf2f8eb 2024-04-23T21:22:06.909 INFO:teuthology.orchestra.run.smithi111.stdout:3874 write 0x2446d5e thru 0x2455e04 (0xf0a7 bytes) 2024-04-23T21:22:06.913 INFO:teuthology.orchestra.run.smithi111.stdout:3875 read 0x3b71b7a thru 0x3b729fa (0xe81 bytes) 2024-04-23T21:22:06.920 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:06.920 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:06.921 INFO:teuthology.orchestra.run.smithi111.stdout:3876 writesame 0xbf76575 thru 0xbf81985 data_size 0x1(0xb411 bytes) 2024-04-23T21:22:06.921 INFO:teuthology.orchestra.run.smithi111.stdout:3877 read 0x32bd710 thru 0x32c73a8 (0x9c99 bytes) 2024-04-23T21:22:06.921 INFO:teuthology.orchestra.run.smithi111.stdout:3878 read 0xbbdade8 thru 0xbbe284e (0x7a67 bytes) 2024-04-23T21:22:06.921 INFO:teuthology.orchestra.run.smithi111.stdout:3879 write 0x6e97ce6 thru 0x6ea5a82 (0xdd9d bytes) 2024-04-23T21:22:06.924 INFO:teuthology.orchestra.run.smithi111.stdout:3880 write 0xa8cf130 thru 0xa8d756f (0x8440 bytes) 2024-04-23T21:22:06.927 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.927 INFO:teuthology.orchestra.run.smithi111.stdout:3882 punch from 0x1dd14 to 0x286e6, (0xa9d2 bytes) 2024-04-23T21:22:06.927 INFO:teuthology.orchestra.run.smithi111.stdout:3884 read 0x1eb4c1d thru 0x1ebe477 (0x985b bytes) 2024-04-23T21:22:06.932 INFO:teuthology.orchestra.run.smithi111.stdout:3885 trunc from 0xbf81986 to 0x869e472 2024-04-23T21:22:06.939 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:06.939 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:06.939 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:06.939 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:06.942 INFO:teuthology.orchestra.run.smithi111.stdout:3886 read 0x4e06ad5 thru 0x4e0eb16 (0x8042 bytes) 2024-04-23T21:22:06.942 INFO:teuthology.orchestra.run.smithi111.stdout:3887 writesame 0x85e38e1 thru 0x85f0aeb data_size 0x1(0xd20b bytes) 2024-04-23T21:22:06.949 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:06.962 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:06.962 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:06.972 INFO:teuthology.orchestra.run.smithi111.stdout:3888 read 0x72bba77 thru 0x72c327e (0x7808 bytes) 2024-04-23T21:22:06.972 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:06.972 INFO:teuthology.orchestra.run.smithi111.stdout:3891 read 0x2710d43 thru 0x271597c (0x4c3a bytes) 2024-04-23T21:22:06.972 INFO:teuthology.orchestra.run.smithi111.stdout:3892 write 0x46e45cb thru 0x46f41b8 (0xfbee bytes) 2024-04-23T21:22:06.975 INFO:teuthology.orchestra.run.smithi111.stdout:3893 write 0x50c8de1 thru 0x50cb85e (0x2a7e bytes) 2024-04-23T21:22:06.983 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:06.984 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:06.988 INFO:teuthology.orchestra.run.smithi111.stdout:3894 writesame 0xd2b1e55 thru 0xd2bb8a8 data_size 0x77(0x9a54 bytes) 2024-04-23T21:22:06.988 INFO:teuthology.orchestra.run.smithi111.stdout:3896 write 0x53ff76d thru 0x540d3ef (0xdc83 bytes) 2024-04-23T21:22:06.991 INFO:teuthology.orchestra.run.smithi111.stdout:3897 clone 31 order 18 su 131072 sc 12 2024-04-23T21:22:06.993 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:06.994+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:07.002 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:07.002 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:07.068 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.068 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.077 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:07.094 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:07.095+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:07.178 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.179 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:07.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:22:07.231 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:07.232+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:07.290 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone30 from 0xd2bb8a9 (overlap 0xaf6179) to 0x693a7b 2024-04-23T21:22:07.296 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:07.297+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:07.298 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.298 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.307 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:07.396 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:07.397+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:07.411 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-04-23T21:22:07.485 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.485 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.497 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:07.497 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:07.498+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:07.598 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:07.599+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:07.645 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.645 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:07.662 INFO:teuthology.orchestra.run.smithi111.stdout:3898 trunc from 0xd2bb8a9 to 0xd29942f 2024-04-23T21:22:07.687 INFO:teuthology.orchestra.run.smithi111.stdout:3899 trunc from 0xd29942f to 0x35b17ff 2024-04-23T21:22:07.698 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:07.699+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:07.709 INFO:teuthology.orchestra.run.smithi111.stdout:3900 trunc from 0x35b17ff to 0x81df65a 2024-04-23T21:22:07.714 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:07.714 INFO:teuthology.orchestra.run.smithi111.stdout:3902 read 0x7f1d4ab thru 0x7f2c8a8 (0xf3fe bytes) 2024-04-23T21:22:07.714 INFO:teuthology.orchestra.run.smithi111.stdout:3903 writesame 0xd0d2e2 thru 0xd13efc data_size 0x2d(0x6c1b bytes) 2024-04-23T21:22:07.718 INFO:teuthology.orchestra.run.smithi111.stdout:3904 read 0x3654134 thru 0x365d19c (0x9069 bytes) 2024-04-23T21:22:07.718 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:07.718 INFO:teuthology.orchestra.run.smithi111.stdout:3907 read 0x4b40c30 thru 0x4b4d6ed (0xcabe bytes) 2024-04-23T21:22:07.721 INFO:teuthology.orchestra.run.smithi111.stdout:3908 trunc from 0x81df65a to 0x52adb0d 2024-04-23T21:22:07.727 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:07.735 INFO:teuthology.orchestra.run.smithi111.stdout:3910 write 0x7470d63 thru 0x747f5a6 (0xe844 bytes) 2024-04-23T21:22:07.745 INFO:teuthology.orchestra.run.smithi111.stdout:3911 trunc from 0x747f5a7 to 0x1a992ef 2024-04-23T21:22:07.760 INFO:teuthology.orchestra.run.smithi111.stdout:3912 writesame 0x29200d thru 0x2a1241 data_size 0x1(0xf235 bytes) 2024-04-23T21:22:07.772 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.772 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.782 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:07.797 INFO:teuthology.orchestra.run.smithi111.stdout:3914 write 0xebd013 thru 0xec044f (0x343d bytes) 2024-04-23T21:22:07.803 INFO:teuthology.orchestra.run.smithi111.stdout:3915 read 0x76e02f thru 0x77ca36 (0xea08 bytes) 2024-04-23T21:22:07.819 INFO:teuthology.orchestra.run.smithi111.stdout:3916 write 0x9384662 thru 0x9392b8c (0xe52b bytes) 2024-04-23T21:22:07.822 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:07.822 INFO:teuthology.orchestra.run.smithi111.stdout:3918 read 0x55f99aa thru 0x5603eb6 (0xa50d bytes) 2024-04-23T21:22:07.827 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:07.828+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:07.834 INFO:teuthology.orchestra.run.smithi111.stdout:3920 write 0xd9f0f9b thru 0xd9fa58f (0x95f5 bytes) 2024-04-23T21:22:07.836 INFO:teuthology.orchestra.run.smithi111.stdout:3922 punch from 0x5d7e1ef to 0x5d7efcc, (0xddd bytes) 2024-04-23T21:22:07.837 INFO:teuthology.orchestra.run.smithi111.stdout:3923 read 0x56cba7b thru 0x56d9d95 (0xe31b bytes) 2024-04-23T21:22:07.837 INFO:teuthology.orchestra.run.smithi111.stdout:3924 punch from 0x3de955f to 0x3de9b76, (0x617 bytes) 2024-04-23T21:22:07.837 INFO:teuthology.orchestra.run.smithi111.stdout:3925 write 0x5057421 thru 0x5064fad (0xdb8d bytes) 2024-04-23T21:22:07.843 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:07.843 INFO:teuthology.orchestra.run.smithi111.stdout:3928 write 0x3d4c110 thru 0x3d4dfde (0x1ecf bytes) 2024-04-23T21:22:07.846 INFO:teuthology.orchestra.run.smithi111.stdout:3929 writesame 0xa577d37 thru 0xa582218 data_size 0x5a(0xa4e2 bytes) 2024-04-23T21:22:07.860 INFO:teuthology.orchestra.run.smithi111.stdout:3931 trunc from 0xd9fa590 to 0x4b30b9d 2024-04-23T21:22:07.879 INFO:teuthology.orchestra.run.smithi111.stdout:3932 writesame 0x51bc858 thru 0x51c27d5 data_size 0x11(0x5f7e bytes) 2024-04-23T21:22:07.885 INFO:teuthology.orchestra.run.smithi111.stdout:3933 trunc from 0x51c27d6 to 0xa0504bd 2024-04-23T21:22:07.889 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:07.889 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:07.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:07.900 INFO:teuthology.orchestra.run.smithi111.stdout:3935 trunc from 0xa0504bd to 0x23a61b2 2024-04-23T21:22:07.912 INFO:teuthology.orchestra.run.smithi111.stdout:3936 punch from 0x16f7727 to 0x1701906, (0xa1df bytes) 2024-04-23T21:22:07.912 INFO:teuthology.orchestra.run.smithi111.stdout:3938 punch from 0x19ad25b to 0x19b15f5, (0x439a bytes) 2024-04-23T21:22:07.917 INFO:teuthology.orchestra.run.smithi111.stdout:3939 writesame 0x251ea53 thru 0x2527579 data_size 0x7(0x8b27 bytes) 2024-04-23T21:22:07.918 INFO:teuthology.orchestra.run.smithi111.stdout:3940 punch from 0x11d83ca to 0x11e30b2, (0xace8 bytes) 2024-04-23T21:22:07.918 INFO:teuthology.orchestra.run.smithi111.stdout:3941 read 0x10062b thru 0x10c00d (0xb9e3 bytes) 2024-04-23T21:22:07.932 INFO:teuthology.orchestra.run.smithi111.stdout:3942 writesame 0x9604b24 thru 0x960a03b data_size 0x1c(0x5518 bytes) 2024-04-23T21:22:07.932 INFO:teuthology.orchestra.run.smithi111.stdout:3943 write 0x482ca37 thru 0x482ef73 (0x253d bytes) 2024-04-23T21:22:07.939 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:07.940+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:07.947 INFO:teuthology.orchestra.run.smithi111.stdout:3945 write 0xded0ddd thru 0xdedf242 (0xe466 bytes) 2024-04-23T21:22:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:3946 trunc from 0xdedf243 to 0x4e8f86b 2024-04-23T21:22:07.981 INFO:teuthology.orchestra.run.smithi111.stdout:3947 write 0x8f17ef0 thru 0x8f1901d (0x112e bytes) 2024-04-23T21:22:07.983 INFO:teuthology.orchestra.run.smithi111.stdout:3948 read 0x6530a82 thru 0x653d2f0 (0xc86f bytes) 2024-04-23T21:22:07.996 INFO:teuthology.orchestra.run.smithi111.stdout:3949 writesame 0xd226db7 thru 0xd235e26 data_size 0x8(0xf070 bytes) 2024-04-23T21:22:07.996 INFO:teuthology.orchestra.run.smithi111.stdout:3950 read 0x47eaa02 thru 0x47ed7da (0x2dd9 bytes) 2024-04-23T21:22:07.996 INFO:teuthology.orchestra.run.smithi111.stdout:3951 write 0x363ab29 thru 0x3643e45 (0x931d bytes) 2024-04-23T21:22:08.002 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:08.002+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:08.003 INFO:teuthology.orchestra.run.smithi111.stdout:3953 read 0x2d00e5f thru 0x2d085bf (0x7761 bytes) 2024-04-23T21:22:08.003 INFO:teuthology.orchestra.run.smithi111.stdout:3954 punch from 0xc495ae0 to 0xc4a3179, (0xd699 bytes) 2024-04-23T21:22:08.003 INFO:teuthology.orchestra.run.smithi111.stdout:3956 writesame 0x84cf229 thru 0x84d4618 data_size 0x4f(0x53f0 bytes) 2024-04-23T21:22:08.011 INFO:teuthology.orchestra.run.smithi111.stdout:3957 writesame 0xdedf1db thru 0xdee872b data_size 0x37(0x9551 bytes) 2024-04-23T21:22:08.016 INFO:teuthology.orchestra.run.smithi111.stdout:3958 trunc from 0xdee872c to 0x9f49ab2 2024-04-23T21:22:08.034 INFO:teuthology.orchestra.run.smithi111.stdout:3959 write 0xd8a68fd thru 0xd8b39a5 (0xd0a9 bytes) 2024-04-23T21:22:08.084 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.084 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:3960 trunc from 0xd8b39a6 to 0x8ff69e3 2024-04-23T21:22:08.085 INFO:teuthology.orchestra.run.smithi111.stdout:3961 trunc from 0x8ff69e3 to 0x2a841a4 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3962 punch from 0x6a1296 to 0x6ad477, (0xc1e1 bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3963 read 0x8573d7 thru 0x859448 (0x2072 bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3966 write 0x19a081b thru 0x19af6db (0xeec1 bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3967 write 0x386398 thru 0x386dc6 (0xa2f bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3970 read 0x1003af5 thru 0x1006580 (0x2a8c bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3972 read 0x37079e thru 0x370fa6 (0x809 bytes) 2024-04-23T21:22:08.086 INFO:teuthology.orchestra.run.smithi111.stdout:3973 read 0x262eff6 thru 0x263281f (0x382a bytes) 2024-04-23T21:22:08.089 INFO:teuthology.orchestra.run.smithi111.stdout:3974 trunc from 0x2a841a4 to 0xc7ce0f6 2024-04-23T21:22:08.095 INFO:teuthology.orchestra.run.smithi111.stdout:3975 writesame 0x313a8dc thru 0x3140010 data_size 0x2f(0x5735 bytes) 2024-04-23T21:22:08.102 INFO:teuthology.orchestra.run.smithi111.stdout:3976 trunc from 0xc7ce0f6 to 0xa02247a 2024-04-23T21:22:08.102 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:08.104+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:08.118 INFO:teuthology.orchestra.run.smithi111.stdout:3978 trunc from 0xa02247a to 0x11908b9 2024-04-23T21:22:08.135 INFO:teuthology.orchestra.run.smithi111.stdout:3980 read 0xb7571b thru 0xb790f0 (0x39d6 bytes) 2024-04-23T21:22:08.147 INFO:teuthology.orchestra.run.smithi111.stdout:3981 writesame 0x4e4a065 thru 0x4e54a3f data_size 0x3b(0xa9db bytes) 2024-04-23T21:22:08.147 INFO:teuthology.orchestra.run.smithi111.stdout:3982 write 0x23f91fd thru 0x23fc82e (0x3632 bytes) 2024-04-23T21:22:08.150 INFO:teuthology.orchestra.run.smithi111.stdout:3983 punch from 0x11be651 to 0x11ce14a, (0xfaf9 bytes) 2024-04-23T21:22:08.157 INFO:teuthology.orchestra.run.smithi111.stdout:3984 write 0x6293d2f thru 0x62960d6 (0x23a8 bytes) 2024-04-23T21:22:08.160 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.171 INFO:teuthology.orchestra.run.smithi111.stdout:3986 writesame 0xa187d23 thru 0xa196d12 data_size 0x2(0xeff0 bytes) 2024-04-23T21:22:08.171 INFO:teuthology.orchestra.run.smithi111.stdout:3987 writesame 0x42764b7 thru 0x4285b5e data_size 0x24(0xf6a8 bytes) 2024-04-23T21:22:08.176 INFO:teuthology.orchestra.run.smithi111.stdout:3988 read 0x775dfd0 thru 0x7767fa6 (0x9fd7 bytes) 2024-04-23T21:22:08.176 INFO:teuthology.orchestra.run.smithi111.stdout:3989 read 0x11a56b9 thru 0x11ae60a (0x8f52 bytes) 2024-04-23T21:22:08.176 INFO:teuthology.orchestra.run.smithi111.stdout:3990 read 0x69f69f6 thru 0x6a04b29 (0xe134 bytes) 2024-04-23T21:22:08.176 INFO:teuthology.orchestra.run.smithi111.stdout:3991 writesame 0x689930f thru 0x689e971 data_size 0x1(0x5663 bytes) 2024-04-23T21:22:08.186 INFO:teuthology.orchestra.run.smithi111.stdout:3993 read 0x8dc2e84 thru 0x8dd2727 (0xf8a4 bytes) 2024-04-23T21:22:08.186 INFO:teuthology.orchestra.run.smithi111.stdout:3994 punch from 0x1d157bf to 0x1d18c52, (0x3493 bytes) 2024-04-23T21:22:08.186 INFO:teuthology.orchestra.run.smithi111.stdout:3996 read 0x3c56939 thru 0x3c61e6a (0xb532 bytes) 2024-04-23T21:22:08.190 INFO:teuthology.orchestra.run.smithi111.stdout:3998 trunc from 0xa196d13 to 0x6d46044 2024-04-23T21:22:08.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:08.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:08.195 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T21:22:08.196 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:22:08.203 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:08.204+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:08.204 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:22:08.215 INFO:teuthology.orchestra.run.smithi111.stdout:4000 writesame 0x9a57caa thru 0x9a58967 data_size 0x7(0xcbe bytes) 2024-04-23T21:22:08.215 INFO:teuthology.orchestra.run.smithi111.stdout:4001 punch from 0x1f15242 to 0x1f249ff, (0xf7bd bytes) 2024-04-23T21:22:08.224 INFO:teuthology.orchestra.run.smithi111.stdout:4002 write 0xb033303 thru 0xb0342b0 (0xfae bytes) 2024-04-23T21:22:08.228 INFO:teuthology.orchestra.run.smithi111.stdout:4003 read 0x542d358 thru 0x5430ac8 (0x3771 bytes) 2024-04-23T21:22:08.232 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.232 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.238 INFO:teuthology.orchestra.run.smithi111.stdout:4004 write 0xbfbbcca thru 0xbfc4c80 (0x8fb7 bytes) 2024-04-23T21:22:08.239 INFO:teuthology.orchestra.run.smithi111.stdout:4005 read 0x9e216de thru 0x9e26af4 (0x5417 bytes) 2024-04-23T21:22:08.239 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.239 INFO:teuthology.orchestra.run.smithi111.stdout:4008 read 0xbdca592 thru 0xbdd10dd (0x6b4c bytes) 2024-04-23T21:22:08.239 INFO:teuthology.orchestra.run.smithi111.stdout:4011 write 0x500e454 thru 0x501cf38 (0xeae5 bytes) 2024-04-23T21:22:08.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:08.243 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.243 INFO:teuthology.orchestra.run.smithi111.stdout:4013 read 0x9157766 thru 0x9160624 (0x8ebf bytes) 2024-04-23T21:22:08.250 INFO:teuthology.orchestra.run.smithi111.stdout:4014 writesame 0xdfc0ed7 thru 0xdfcfa4c data_size 0x1(0xeb76 bytes) 2024-04-23T21:22:08.250 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.250 INFO:teuthology.orchestra.run.smithi111.stdout:4016 read 0x14ad32 thru 0x14dfb9 (0x3288 bytes) 2024-04-23T21:22:08.251 INFO:teuthology.orchestra.run.smithi111.stdout:4020 write 0x1c2c135 thru 0x1c3bb47 (0xfa13 bytes) 2024-04-23T21:22:08.254 INFO:teuthology.orchestra.run.smithi111.stdout:4022 write 0xb6ed1f3 thru 0xb6fcac8 (0xf8d6 bytes) 2024-04-23T21:22:08.258 INFO:teuthology.orchestra.run.smithi111.stdout:4023 write 0xad1c564 thru 0xad200af (0x3b4c bytes) 2024-04-23T21:22:08.264 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.264 INFO:teuthology.orchestra.run.smithi111.stdout:4026 punch from 0x3e5bbfb to 0x3e674e4, (0xb8e9 bytes) 2024-04-23T21:22:08.264 INFO:teuthology.orchestra.run.smithi111.stdout:4028 writesame 0x2158fe5 thru 0x215abcc data_size 0x2f(0x1be8 bytes) 2024-04-23T21:22:08.267 INFO:teuthology.orchestra.run.smithi111.stdout:4029 write 0x7e9df17 thru 0x7ea594c (0x7a36 bytes) 2024-04-23T21:22:08.274 INFO:teuthology.orchestra.run.smithi111.stdout:4030 writesame 0x7307c39 thru 0x730bf59 data_size 0x7(0x4321 bytes) 2024-04-23T21:22:08.278 INFO:teuthology.orchestra.run.smithi111.stdout:4031 read 0x8f2f1dc thru 0x8f32150 (0x2f75 bytes) 2024-04-23T21:22:08.278 INFO:teuthology.orchestra.run.smithi111.stdout:4032 read 0xc3fea5 thru 0xc44dae (0x4f0a bytes) 2024-04-23T21:22:08.278 INFO:teuthology.orchestra.run.smithi111.stdout:4033 write 0x8bad35f thru 0x8bb210c (0x4dae bytes) 2024-04-23T21:22:08.281 INFO:teuthology.orchestra.run.smithi111.stdout:4034 punch from 0x32e0b21 to 0x32e4ded, (0x42cc bytes) 2024-04-23T21:22:08.282 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.282 INFO:teuthology.orchestra.run.smithi111.stdout:4038 writesame 0xb526b4e thru 0xb533731 data_size 0x2(0xcbe4 bytes) 2024-04-23T21:22:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:4039 read 0xdf1bdcc thru 0xdf27f2b (0xc160 bytes) 2024-04-23T21:22:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:4040 punch from 0x88c6570 to 0x88d436a, (0xddfa bytes) 2024-04-23T21:22:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:4041 read 0xdf7181 thru 0xdfbfa5 (0x4e25 bytes) 2024-04-23T21:22:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:4042 punch from 0x14b7f09 to 0x14c260c, (0xa703 bytes) 2024-04-23T21:22:08.297 INFO:teuthology.orchestra.run.smithi111.stdout:4044 read 0x601540 thru 0x610849 (0xf30a bytes) 2024-04-23T21:22:08.298 INFO:teuthology.orchestra.run.smithi111.stdout:4045 read 0xa7d797e thru 0xa7e6ccd (0xf350 bytes) 2024-04-23T21:22:08.298 INFO:teuthology.orchestra.run.smithi111.stdout:4046 writesame 0xcef6262 thru 0xcf015ee data_size 0x1d(0xb38d bytes) 2024-04-23T21:22:08.304 INFO:teuthology.orchestra.run.smithi111.stdout:4047 trunc from 0xdfcfa4d to 0xd040f8f 2024-04-23T21:22:08.318 INFO:teuthology.orchestra.run.smithi111.stdout:4048 writesame 0xe3763f0 thru 0xe382543 data_size 0x2(0xc154 bytes) 2024-04-23T21:22:08.319 INFO:teuthology.orchestra.run.smithi111.stdout:4051 write 0x486a0d6 thru 0x486a22f (0x15a bytes) 2024-04-23T21:22:08.322 INFO:teuthology.orchestra.run.smithi111.stdout:4052 write 0x4422b9c thru 0x4425ea1 (0x3306 bytes) 2024-04-23T21:22:08.326 INFO:teuthology.orchestra.run.smithi111.stdout:4053 read 0x57a7277 thru 0x57a8b2a (0x18b4 bytes) 2024-04-23T21:22:08.326 INFO:teuthology.orchestra.run.smithi111.stdout:4054 read 0x1283f02 thru 0x12845bb (0x6ba bytes) 2024-04-23T21:22:08.327 INFO:teuthology.orchestra.run.smithi111.stdout:4055 read 0xd979313 thru 0xd985c14 (0xc902 bytes) 2024-04-23T21:22:08.327 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:08.328+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:08.342 INFO:teuthology.orchestra.run.smithi111.stdout:4056 trunc from 0xe382544 to 0x70373c 2024-04-23T21:22:08.374 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:08.374 INFO:teuthology.orchestra.run.smithi111.stdout:4058 clone 32 order 22 su 131072 sc 5 2024-04-23T21:22:08.394 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.394 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.403 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:08.405 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:08.406+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:08.506 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:08.508+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:08.512 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:22:08.514+0000 7ff439881640 0 --2- 172.21.15.111:0/2618283253 >> v2:172.21.15.111:3300/0 conn(0x7ff4340a9850 0x7ff43414dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-23T21:22:08.545 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.546 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:08.606 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:08.607+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:08.657 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.657 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 64, 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:22:08.201663+0000", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:21:48.929598+0000", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:20:59.765723+0000", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5806080, 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "nodeep-scrub", 2024-04-23T21:22:08.701 INFO:teuthology.orchestra.run.smithi111.stdout: "noscrub", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 5, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:22:08.702 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "luminous", 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:08.703 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:08.704 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:08.705 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:08.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:08.707 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 3, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 3, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 1, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 3, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 3, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.708 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:08.709 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:22:08.710 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:08.711 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:08.712 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:22:08.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 8, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 8, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.25, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:08.715 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "63", 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:08.716 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 33, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 63, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:08.717 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:08.718 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 1.75, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 1.75, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 1, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.75, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.75, 2024-04-23T21:22:08.719 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.04681396484375, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.720 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:22:08.721 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:22:08.722 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.725 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 13, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 17, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 45, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 48, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.726 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.728 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.729 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/723341542", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/723341542", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/723341542", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/723341542", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.730 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:22:08.733 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:22:08.734 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:22:08.735 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:22:08.736 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:22:08.739 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:08.740 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 48, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.744 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.745 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 29, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.750 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 18, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.751 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:22:08.755 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 59, 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.759 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.760 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.761 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.762 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.763 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.764 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:08.766 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [ 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.6", 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "mappings": [ 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "from": 6, 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: "to": 7 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.767 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [], 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:22:08.768 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [], 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [], 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [], 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:08.769 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:08.772 INFO:tasks.thrashosds.thrasher:j is {'epoch': 64, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:22:08.201663+0000', 'last_up_change': '2024-04-23T21:21:48.929598+0000', 'last_in_change': '2024-04-23T21:20:59.765723+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.04681396484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 17, 'up_from': 45, 'up_thru': 48, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 723341542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 723341542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 723341542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 723341542}]}, 'public_addr': '172.21.15.111:6817/723341542', 'cluster_addr': '172.21.15.111:6819/723341542', 'heartbeat_back_addr': '172.21.15.111:6823/723341542', 'heartbeat_front_addr': '172.21.15.111:6821/723341542', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 29, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 18, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 59, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.6', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:22:08.772 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-04-23T21:22:08.772 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:08.772+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:08.871 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:08.872+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:08.872 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.873 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.881 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:08.972 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:08.973+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:08.983 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:08.983 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:08.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:09.014 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:09.076 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:09.077+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:09.133 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:09.133 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:09.135 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:09.136 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:09.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:09.150 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:09.150 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:09.166 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:09.167 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:09.174 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:09.175+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:09.183 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:09.183 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:09.200 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:09.200 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:09.220 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:09.221 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:09.242 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:09.242 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:09.261 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:09.261 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:09.275 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:09.276+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:09.295 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:09.295 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:09.295 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:09.295 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:09.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:09.311 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone31 from 0x70373c (overlap 0x70373c) to 0x435457 2024-04-23T21:22:09.388 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:09.389+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:09.448 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-04-23T21:22:09.451 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:09.451 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:09.451 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:09.451 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:09.451 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:09.462 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:09.479 INFO:teuthology.orchestra.run.smithi111.stdout:4061 write 0x1edaafe thru 0x1ee3f38 (0x943b bytes) 2024-04-23T21:22:09.483 INFO:teuthology.orchestra.run.smithi111.stdout:4063 punch from 0x1e9a927 to 0x1e9d24d, (0x2926 bytes) 2024-04-23T21:22:09.503 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:09.504+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:09.504 INFO:teuthology.orchestra.run.smithi111.stdout:4064 write 0xdd63bf6 thru 0xdd70fe2 (0xd3ed bytes) 2024-04-23T21:22:09.514 INFO:teuthology.orchestra.run.smithi111.stdout:4065 write 0xe8a883b thru 0xe8b84e2 (0xfca8 bytes) 2024-04-23T21:22:09.515 INFO:teuthology.orchestra.run.smithi111.stdout:4068 clone 33 order 24 su 524288 sc 8 2024-04-23T21:22:09.578 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:09.578+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:09.606 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:09.606 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:09.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:09.606 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:09.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3199 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3047424~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185641263000001, 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024568476999999998, 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3199 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420574+0000", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420575+0000", 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420584+0000", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7320000000000004e-06 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420590+0000", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5459999999999997e-06 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420595+0000", 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6619999999999996e-06 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.444956+0000", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024361108999999999 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445039+0000", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2669000000000004e-05 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445143+0000", 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104361 2024-04-23T21:22:09.610 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3200 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3112960~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.1856139, 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024649676999999998, 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3200 2024-04-23T21:22:09.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420602+0000", 2024-04-23T21:22:09.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420604+0000", 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.289e-06 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420610+0000", 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1170000000000003e-06 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420612+0000", 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7870000000000002e-06 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445160+0000", 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02454781 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445198+0000", 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7892000000000002e-05 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445251+0000", 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3389e-05 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3201 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3178496~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185599650999997, 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024845473, 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.615 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3201 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:09.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420616+0000", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2600000000000001e-07 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420618+0000", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.237e-06 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420622+0000", 2024-04-23T21:22:09.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6999999999999999e-06 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420632+0000", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445285+0000", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024653337000000001 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445361+0000", 2024-04-23T21:22:09.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5883000000000005e-05 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445462+0000", 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100597 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3202 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3244032~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185577408, 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025036078, 2024-04-23T21:22:09.619 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3202 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420638+0000", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420639+0000", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420641+0000", 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0080000000000001e-06 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420646+0000", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3700000000000003e-06 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420650+0000", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0419999999999996e-06 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445504+0000", 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024854459999999998 2024-04-23T21:22:09.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445574+0000", 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9853999999999999e-05 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445674+0000", 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100023 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3203 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3309568~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.623 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185511922000003, 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025140824999999999, 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3203 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420704+0000", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2600000000000002e-07 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420707+0000", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9050000000000001e-06 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420715+0000", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9529999999999999e-06 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420721+0000", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4410000000000001e-06 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445713+0000", 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024992593 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445769+0000", 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5442999999999998e-05 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445845+0000", 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5864000000000004e-05 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3204 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3375104~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185486179999998, 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025304432000000002, 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3204 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420730+0000", 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420734+0000", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.155e-06 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420742+0000", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5870000000000004e-06 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420746+0000", 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6949999999999999e-06 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445867+0000", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02512089 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.445940+0000", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2317000000000006e-05 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446034+0000", 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4340999999999997e-05 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.631 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3205 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3440640~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185462170999998, 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025527610999999999, 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3205 2024-04-23T21:22:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420754+0000", 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:09.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420756+0000", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2029999999999999e-06 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420763+0000", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9090000000000003e-06 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420768+0000", 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8110000000000002e-06 2024-04-23T21:22:09.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446077+0000", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025308634 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446175+0000", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8412000000000006e-05 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446281+0000", 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000106161 2024-04-23T21:22:09.635 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3206 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3506176~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185439612000003, 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025639889999999999, 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3206 2024-04-23T21:22:09.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420776+0000", 2024-04-23T21:22:09.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420779+0000", 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420785+0000", 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.032e-06 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420789+0000", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7090000000000001e-06 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446313+0000", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025523679000000001 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446356+0000", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3214000000000003e-05 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446416+0000", 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9614000000000002e-05 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3207 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3571712~32768] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185419809000003, 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025857939, 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3207 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420796+0000", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420798+0000", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2299999999999998e-06 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420804+0000", 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7749999999999998e-06 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420809+0000", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5480000000000003e-06 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446452+0000", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025643351000000002 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446549+0000", 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7021999999999994e-05 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446654+0000", 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104627 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3208 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3637248~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185400323000003, 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025932411999999998, 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.644 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3208 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420815+0000", 2024-04-23T21:22:09.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420816+0000", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420818+0000", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420824+0000", 2024-04-23T21:22:09.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9989999999999997e-06 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420829+0000", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446667+0000", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025838310999999999 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446708+0000", 2024-04-23T21:22:09.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0880000000000002e-05 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446748+0000", 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9776e-05 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3209 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3702784~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185380524000003, 2024-04-23T21:22:09.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026043803000000001, 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3209 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420835+0000", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420836+0000", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420838+0000", 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420844+0000", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0780000000000001e-06 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420849+0000", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9609999999999996e-06 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446786+0000", 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025937584 2024-04-23T21:22:09.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446832+0000", 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.621e-05 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446879+0000", 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6588000000000002e-05 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.652 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3210 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3768320~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185360822, 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026132595000000002, 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3210 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:09.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420855+0000", 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420857+0000", 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420863+0000", 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420868+0000", 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4859999999999999e-06 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446902+0000", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026034399999999999 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446947+0000", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5009000000000002e-05 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.446987+0000", 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0549000000000001e-05 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3211 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3833856~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185304696000003, 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026185706999999999, 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3211 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420911+0000", 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420912+0000", 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5000000000000003e-07 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420918+0000", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8709999999999999e-06 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420934+0000", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5885000000000001e-05 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420940+0000", 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8100000000000003e-06 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447012+0000", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026071836000000001 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447053+0000", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1461000000000002e-05 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447097+0000", 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3294000000000003e-05 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3212 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3899392~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185267475000003, 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026246375999999998, 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3212 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420948+0000", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420949+0000", 2024-04-23T21:22:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420951+0000", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5299999999999999e-06 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420957+0000", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2419999999999997e-06 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420963+0000", 2024-04-23T21:22:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9909999999999999e-06 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447119+0000", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026155800999999999 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447157+0000", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7759999999999998e-05 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447195+0000", 2024-04-23T21:22:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7582e-05 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3213 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 3964928~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185217162000001, 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026298048000000001, 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3213 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.420999+0000", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421003+0000", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4170000000000001e-06 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421013+0000", 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0322e-05 2024-04-23T21:22:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421021+0000", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2730000000000002e-06 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447217+0000", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026195693999999999 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447257+0000", 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0088e-05 2024-04-23T21:22:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447297+0000", 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9635000000000003e-05 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3214 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4030464~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185184266999997, 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026363471999999999, 2024-04-23T21:22:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3214 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421031+0000", 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421032+0000", 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421034+0000", 2024-04-23T21:22:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4729999999999999e-06 2024-04-23T21:22:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421041+0000", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.657e-06 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421047+0000", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447318+0000", 2024-04-23T21:22:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026271402999999999 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447356+0000", 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7981000000000001e-05 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447395+0000", 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8954e-05 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3215 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4096000~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185161962999999, 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026442377, 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3215 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421054+0000", 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421056+0000", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9750000000000001e-06 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421063+0000", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421068+0000", 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1420000000000001e-06 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447416+0000", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026348292999999998 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447456+0000", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9885000000000002e-05 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447496+0000", 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0198999999999999e-05 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:3216 3.f 3:f1c77f3f:::rbd_data.1118c9f76dd8.0000000000000006:4 [read 4161536~0] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e23)", 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.185141547000001, 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026510259000000001, 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 3216 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421074+0000", 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1399999999999998e-07 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421076+0000", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8589999999999999e-06 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421083+0000", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3380000000000001e-06 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.421088+0000", 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447508+0000", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026420815 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447547+0000", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8177999999999999e-05 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:24.447584+0000", 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7857e-05 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61148 3.13 3:cc4c0078:::rbd_data.1118e08f6c14.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.2134018649999998, 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.033335561999999999, 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61148 2024-04-23T21:22:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392814+0000", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.392816+0000", 2024-04-23T21:22:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0609999999999998e-06 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407607+0000", 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014790792000000001 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407632+0000", 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5092999999999998e-05 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407646+0000", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4134999999999999e-05 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407719+0000", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3206000000000003e-05 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.407831+0000", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111797 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.408130+0000", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00029870400000000001 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424207+0000", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016077281999999998 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426036+0000", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001829378 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426098+0000", 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1693999999999996e-05 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.426149+0000", 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1419999999999999e-05 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4.4787710330000001, 2024-04-23T21:22:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:09.695 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:09.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:09.697 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:09.696+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:09.797 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:09.798+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:09.799 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:09.799 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:09.799 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:09.808 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:09.898 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:09.898+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:09.909 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:09.909 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:09.910 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:09.910 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:09.910 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:09.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:10.003 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:10.003+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:10.063 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 21.122726447000002, 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.064 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:10.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:10.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.068 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53205 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 2932736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.278097888000001, 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013002390000000001, 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53205 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785414+0000", 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7599999999999997e-07 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785417+0000", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0019999999999998e-06 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785427+0000", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5929999999999995e-06 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785441+0000", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4764e-05 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798225+0000", 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012784007 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798310+0000", 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4618999999999997e-05 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798416+0000", 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010582900000000001 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53206 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 3309568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.278061287, 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013168311, 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.073 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53206 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785451+0000", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785453+0000", 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:22:10.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785460+0000", 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.742e-06 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785464+0000", 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5390000000000001e-06 2024-04-23T21:22:10.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798454+0000", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012989908999999999 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798540+0000", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6206999999999996e-05 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798618+0000", 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7956999999999999e-05 2024-04-23T21:22:10.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53258 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 753664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277798122, 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013389243, 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53258 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785714+0000", 2024-04-23T21:22:10.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6400000000000001e-07 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785724+0000", 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8560000000000002e-06 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785736+0000", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2339e-05 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785744+0000", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8050000000000007e-06 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798657+0000", 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012913345999999999 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798736+0000", 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8627000000000005e-05 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799102+0000", 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000366406 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.083 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53259 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277755386999999, 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013579291, 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53259 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785757+0000", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785759+0000", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.481e-06 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785769+0000", 2024-04-23T21:22:10.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0234000000000001e-05 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785775+0000", 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.648e-06 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799145+0000", 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013370253 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799229+0000", 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3820999999999996e-05 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799335+0000", 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000106226 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53260 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277728222, 2024-04-23T21:22:10.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01398165, 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53260 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785784+0000", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.27e-07 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785786+0000", 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2520000000000002e-06 2024-04-23T21:22:10.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785792+0000", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6510000000000004e-06 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785798+0000", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3170000000000001e-06 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799366+0000", 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013568573 2024-04-23T21:22:10.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799453+0000", 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6549000000000001e-05 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799765+0000", 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031188100000000002 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.092 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53261 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277706904999999, 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014180051000000001, 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53261 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785805+0000", 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785807+0000", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0190000000000001e-06 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785814+0000", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785820+0000", 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1279999999999996e-06 2024-04-23T21:22:10.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799793+0000", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013973167 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799884+0000", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1049000000000002e-05 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799984+0000", 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010002500000000001 2024-04-23T21:22:10.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53262 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1671168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277683688, 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014362929, 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.097 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53262 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785830+0000", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.131e-06 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785837+0000", 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.849e-06 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785842+0000", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3859999999999998e-06 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800015+0000", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014173008000000001 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800098+0000", 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2268999999999994e-05 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800191+0000", 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2818000000000005e-05 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53263 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1851392~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277661864999999, 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014770998, 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.101 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53263 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785850+0000", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6199999999999998e-07 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785852+0000", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.046e-06 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785858+0000", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4799999999999998e-06 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785864+0000", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1279999999999999e-06 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800236+0000", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014372433 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800327+0000", 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0816000000000006e-05 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800620+0000", 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00029363300000000001 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53264 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:10.105 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277641437, 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014801979999999999, 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53264 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785873+0000", 2024-04-23T21:22:10.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9160000000000001e-06 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785880+0000", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6599999999999998e-06 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785885+0000", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.113e-06 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800660+0000", 2024-04-23T21:22:10.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775352 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800672+0000", 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1528e-05 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53265 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277609344999998, 2024-04-23T21:22:10.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014804595, 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53265 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785905+0000", 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4250000000000001e-06 2024-04-23T21:22:10.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785912+0000", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3889999999999999e-06 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785924+0000", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1953999999999999e-05 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800700+0000", 2024-04-23T21:22:10.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775307 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800707+0000", 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0489999999999998e-06 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53266 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277560468000001, 2024-04-23T21:22:10.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014787503, 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53266 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9999999999999997e-07 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785954+0000", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.649e-06 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785969+0000", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5058999999999999e-05 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785975+0000", 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9529999999999996e-06 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800730+0000", 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014755111 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800738+0000", 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1310000000000006e-06 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53267 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.117 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277527505999998, 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014785247, 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53267 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2e-07 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785992+0000", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8059999999999995e-06 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785999+0000", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3520000000000001e-06 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786006+0000", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5549999999999999e-06 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800760+0000", 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014754259 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800769+0000", 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.755e-06 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53268 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277491259000001, 2024-04-23T21:22:10.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014799623, 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53268 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786021+0000", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786023+0000", 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5569999999999998e-06 2024-04-23T21:22:10.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786031+0000", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.644e-06 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786036+0000", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4890000000000003e-06 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800808+0000", 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014771467 2024-04-23T21:22:10.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800820+0000", 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1924e-05 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53269 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.277461520999999, 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014798240000000001, 2024-04-23T21:22:10.125 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53269 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786057+0000", 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3040000000000001e-06 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786065+0000", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8640000000000002e-06 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786077+0000", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1595e-05 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800840+0000", 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014762542999999999 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800848+0000", 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4249999999999998e-06 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:61142 3.4 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 9.6853577909999995, 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.019170651, 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.129 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 61142 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:10.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378156+0000", 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2989999999999999e-06 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383079+0000", 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049228930000000002 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383111+0000", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2338000000000001e-05 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383125+0000", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4039999999999999e-05 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383226+0000", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010133599999999999 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383338+0000", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000111777 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384242+0000", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00090330900000000003 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.395145+0000", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010903042999999999 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:10.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397245+0000", 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002099952 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397289+0000", 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4792000000000001e-05 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397324+0000", 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4872000000000001e-05 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.134 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63700 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.8943209579999998, 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027843201000000001, 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63700 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.135 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169192+0000", 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5069999999999999e-06 2024-04-23T21:22:10.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169200+0000", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1429999999999998e-06 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169214+0000", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4422e-05 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169221+0000", 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6699999999999997e-06 2024-04-23T21:22:10.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169280+0000", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.906e-05 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169343+0000", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2478999999999996e-05 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.174410+0000", 2024-04-23T21:22:10.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005067078 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.195126+0000", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020716472 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196931+0000", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018052929999999999 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196984+0000", 2024-04-23T21:22:10.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2627999999999998e-05 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.197034+0000", 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9449e-05 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63757 3.19 3:9ccb22ce:::rbd_data.1118139b0f86.0000000000000025:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.775995054, 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.015960412, 2024-04-23T21:22:10.140 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63757 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287517+0000", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1900000000000001e-07 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287522+0000", 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287533+0000", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.079e-05 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287540+0000", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2019999999999998e-06 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287617+0000", 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7014000000000005e-05 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287667+0000", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9421e-05 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 3", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.290130+0000", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0024636150000000002 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303396+0000", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013265476 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:10.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303410+0000", 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3946e-05 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303446+0000", 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.612e-05 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303477+0000", 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0839000000000001e-05 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63940 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.5908944979999999, 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.016049744000000001, 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63940 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472618+0000", 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.61e-07 2024-04-23T21:22:10.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472622+0000", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1690000000000002e-06 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472632+0000", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0399e-05 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472637+0000", 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2310000000000004e-06 2024-04-23T21:22:10.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472674+0000", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6465999999999999e-05 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472718+0000", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580999999999999e-05 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.475549+0000", 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028307340000000001 2024-04-23T21:22:10.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488593+0000", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013043406 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488604+0000", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1649e-05 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488636+0000", 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1622999999999999e-05 2024-04-23T21:22:10.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488667+0000", 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0725000000000001e-05 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:64217 3.6 3:67a8ca51:::rbd_data.1118139b0f86.0000000000000049:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.2469559390000002, 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013303083, 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.151 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 64217 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:10.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816556+0000", 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2500000000000005e-07 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816560+0000", 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2030000000000002e-06 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816571+0000", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0514e-05 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816576+0000", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9169999999999997e-06 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816618+0000", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2190000000000001e-05 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816710+0000", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2008000000000004e-05 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.818915+0000", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002205046 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829742+0000", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010826987 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:10.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829794+0000", 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2126000000000002e-05 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829828+0000", 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4406999999999998e-05 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829858+0000", 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9960000000000001e-05 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.156 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.157 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.158 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:10.158+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:10.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:10.220 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:22:10.258 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:10.259+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:10.310 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.310 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:10.310 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:10.311 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.321 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:10.337 INFO:teuthology.orchestra.run.smithi111.stdout:flattening image image_client.0-clone32 2024-04-23T21:22:10.359 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:10.359+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:10.434 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.434 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:10.435 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:10.435 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:10.435 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.444 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:10.460 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:10.461+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:10.560 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:10.561+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:10.562 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-04-23T21:22:10.574 INFO:teuthology.orchestra.run.smithi111.stdout:4070 read 0xa935e6a thru 0xa93ade8 (0x4f7f bytes) 2024-04-23T21:22:10.575 INFO:teuthology.orchestra.run.smithi111.stdout:4072 read 0x6d4f46a thru 0x6d51098 (0x1c2f bytes) 2024-04-23T21:22:10.576 INFO:teuthology.orchestra.run.smithi111.stdout:4073 read 0xe59524e thru 0xe59d890 (0x8643 bytes) 2024-04-23T21:22:10.576 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.576 INFO:teuthology.orchestra.run.smithi111.stdout:4076 read 0x95d7284 thru 0x95df17a (0x7ef7 bytes) 2024-04-23T21:22:10.577 INFO:teuthology.orchestra.run.smithi111.stdout:4077 writesame 0x144146e thru 0x144b71e data_size 0x1(0xa2b1 bytes) 2024-04-23T21:22:10.598 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.598 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:10.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:10.598 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:10.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.168053964000002, 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:10.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:10.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:10.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:10.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.168031153000001, 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.168011542999999, 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.607 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:10.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:10.611 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167992679999998, 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:10.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:10.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:10.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.615 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167973385000003, 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.616 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:10.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:10.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:10.619 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167946823999998, 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:10.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:10.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:10.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:10.623 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:10.624 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.624 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.624 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.624 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167905593999997, 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.625 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:10.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:10.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:10.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167871867000002, 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.629 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:10.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167848458000002, 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:10.633 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167815208, 2024-04-23T21:22:10.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:10.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.641 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167791815000001, 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:10.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:10.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:10.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167771266000003, 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.167745506000003, 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:10.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:10.651 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:10.652 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:10.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.16690749, 2024-04-23T21:22:10.654 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.655 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:10.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:10.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.166884158999999, 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.659 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.662 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.165996866, 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.663 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.666 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.165763286000001, 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:10.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:10.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:10.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.165646588000001, 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.671 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.165272174000002, 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.675 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:10.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.163915289000002, 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:10.683 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.684 INFO:teuthology.orchestra.run.smithi111.stdout:4078 write 0x644a7c7 thru 0x6452509 (0x7d43 bytes) 2024-04-23T21:22:10.685 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4080 write 0x9ae354b thru 0x9ae87e9 (0x529f bytes) 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4081 read 0xa932d08 thru 0xa933400 (0x6f9 bytes) 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4084 write 0xd4e3eb7 thru 0xd4e4351 (0x49b bytes) 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4088 read 0x88fbc89 thru 0x890768a (0xba02 bytes) 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4089 write 0x8401d98 thru 0x840cff9 (0xb262 bytes) 2024-04-23T21:22:10.687 INFO:teuthology.orchestra.run.smithi111.stdout:4092 punch from 0x1b28468 to 0x1b31425, (0x8fbd bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4093 punch from 0x8c9fa38 to 0x8ca1626, (0x1bee bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4094 writesame 0xc410ec6 thru 0xc41708f data_size 0x1(0x61ca bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4095 write 0x49cc4ec thru 0x49d1557 (0x506c bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4096 write 0xd46522d thru 0xd46c8f6 (0x76ca bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4097 writesame 0x919ce94 thru 0x91a27f0 data_size 0x1(0x595d bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4099 read 0xd9f121a thru 0xd9f4018 (0x2dff bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4100 read 0xbf29df9 thru 0xbf2e099 (0x42a1 bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4101 read 0x834dcce thru 0x8351b18 (0x3e4b bytes) 2024-04-23T21:22:10.688 INFO:teuthology.orchestra.run.smithi111.stdout:4102 write 0xc188d11 thru 0xc18f074 (0x6364 bytes) 2024-04-23T21:22:10.688 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:10.685+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:10.704 INFO:teuthology.orchestra.run.smithi111.stdout:4103 writesame 0x8b9c89b thru 0x8ba4631 data_size 0x7(0x7d97 bytes) 2024-04-23T21:22:10.709 INFO:teuthology.orchestra.run.smithi111.stdout:4104 writesame 0xe68f25 thru 0xe77fc2 data_size 0x13(0xf09e bytes) 2024-04-23T21:22:10.718 INFO:teuthology.orchestra.run.smithi111.stdout:4105 write 0x950286c thru 0x950ef76 (0xc70b bytes) 2024-04-23T21:22:10.722 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.722 INFO:teuthology.orchestra.run.smithi111.stdout:4108 read 0x6e36d6d thru 0x6e3e27f (0x7513 bytes) 2024-04-23T21:22:10.722 INFO:teuthology.orchestra.run.smithi111.stdout:4110 punch from 0x6d6cfdc to 0x6d73865, (0x6889 bytes) 2024-04-23T21:22:10.723 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.728 INFO:teuthology.orchestra.run.smithi111.stdout:4114 trunc from 0xe8b84e3 to 0x9d1cf93 2024-04-23T21:22:10.739 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.739 INFO:teuthology.orchestra.run.smithi111.stdout:4116 read 0x1885367 thru 0x189007e (0xad18 bytes) 2024-04-23T21:22:10.739 INFO:teuthology.orchestra.run.smithi111.stdout:4118 writesame 0x5f0725f thru 0x5f10377 data_size 0x5f(0x9119 bytes) 2024-04-23T21:22:10.747 INFO:teuthology.orchestra.run.smithi111.stdout:4123 write 0x91b5cad thru 0x91baa76 (0x4dca bytes) 2024-04-23T21:22:10.748 INFO:teuthology.orchestra.run.smithi111.stdout:4125 write 0x13645eb thru 0x137210d (0xdb23 bytes) 2024-04-23T21:22:10.759 INFO:teuthology.orchestra.run.smithi111.stdout:4126 writesame 0xd64eac5 thru 0xd657f59 data_size 0x1f(0x9495 bytes) 2024-04-23T21:22:10.760 INFO:teuthology.orchestra.run.smithi111.stdout:4128 read 0x9918b89 thru 0x9927f3a (0xf3b2 bytes) 2024-04-23T21:22:10.761 INFO:teuthology.orchestra.run.smithi111.stdout:4129 punch from 0xd46be65 to 0xd46d453, (0x15ee bytes) 2024-04-23T21:22:10.761 INFO:teuthology.orchestra.run.smithi111.stdout:4130 writesame 0x264a74d thru 0x264ea8e data_size 0x1(0x4342 bytes) 2024-04-23T21:22:10.770 INFO:teuthology.orchestra.run.smithi111.stdout:4132 punch from 0x21d41b5 to 0x21e1b4e, (0xd999 bytes) 2024-04-23T21:22:10.770 INFO:teuthology.orchestra.run.smithi111.stdout:4133 punch from 0xd0c3873 to 0xd0d2d80, (0xf50d bytes) 2024-04-23T21:22:10.770 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:10.776 INFO:teuthology.orchestra.run.smithi111.stdout:4138 write 0xebaf0f4 thru 0xebbdfc7 (0xeed4 bytes) 2024-04-23T21:22:10.778 INFO:teuthology.orchestra.run.smithi111.stdout:4139 punch from 0xaac7d21 to 0xaad0e82, (0x9161 bytes) 2024-04-23T21:22:10.778 INFO:teuthology.orchestra.run.smithi111.stdout:4141 writesame 0x3697987 thru 0x36a6480 data_size 0x13(0xeafa bytes) 2024-04-23T21:22:10.786 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:10.786+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:10.786 INFO:teuthology.orchestra.run.smithi111.stdout:4142 writesame 0xcfb5a0d thru 0xcfbb2c6 data_size 0x29(0x58ba bytes) 2024-04-23T21:22:10.788 INFO:teuthology.orchestra.run.smithi111.stdout:4145 punch from 0xc6d0058 to 0xc6d3f08, (0x3eb0 bytes) 2024-04-23T21:22:10.788 INFO:teuthology.orchestra.run.smithi111.stdout:4147 clone 34 order 25 su 16777216 sc 12 2024-04-23T21:22:10.802 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.802 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:10.802 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:10.802 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.811 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:10.887 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:10.887+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:10.952 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:10.953 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:10.953 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:10.953 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:10.953 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:10.961 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:11.005 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:11.005+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:11.064 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:11.064 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:11.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:11.064 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:11.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833439791, 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.068 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833410124, 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.069 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833390716, 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.073 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833368908000001, 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.077 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:11.080 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833349304999999, 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:11.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:11.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:11.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.8333297, 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833308858999999, 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:11.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:11.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833279594, 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.093 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:11.097 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.833248400999999, 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:11.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:11.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.025165390000002, 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.102 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:11.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:11.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:11.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021937172000001, 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.106 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:11.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:11.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021914791, 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:11.110 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.111 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:11.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021895513, 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.115 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:11.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021878079, 2024-04-23T21:22:11.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:11.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:11.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.123 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021860628999999, 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:11.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021843254, 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.128 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:11.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:11.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:11.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021825212, 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.132 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:11.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.021800090999999, 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.136 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 23.019680519000001, 2024-04-23T21:22:11.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:11.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:11.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.144 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 10.680057528000001, 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:11.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.148 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.149 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:11.149 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:11.149 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:11.150 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:11.151 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:11.150+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:11.224 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:22:11.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:11.281 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:11.281+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:11.287 INFO:teuthology.orchestra.run.smithi111.stdout:flattening image image_client.0-clone33 2024-04-23T21:22:11.309 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:11.309 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:11.309 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:11.309 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:11.319 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:11.351 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:11.352+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:11.411 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:11.411 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:11.429 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:11.429 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:11.449 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:11.449 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:11.451 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:11.452+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:11.463 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-04-23T21:22:11.468 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:11.468 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:11.476 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:11.476 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:11.487 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:11.487 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:11.488 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:11.505 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:11.506 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:11.525 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:11.525 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:11.547 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:11.547 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:11.552 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:11.552+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:11.619 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:11.619 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:11.628 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:11.653 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:11.653+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:11.732 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:11.732 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:11.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:11.742 INFO:teuthology.orchestra.run.smithi111.stdout:4148 trunc from 0xebbdfc8 to 0x47632ec 2024-04-23T21:22:11.784 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:11.785+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:11.789 INFO:teuthology.orchestra.run.smithi111.stdout:4149 write 0x91c90d1 thru 0x91d3737 (0xa667 bytes) 2024-04-23T21:22:11.792 INFO:teuthology.orchestra.run.smithi111.stdout:4150 read 0x3f5d43f thru 0x3f619e3 (0x45a5 bytes) 2024-04-23T21:22:11.792 INFO:teuthology.orchestra.run.smithi111.stdout:4151 read 0x33faedd thru 0x3403ebc (0x8fe0 bytes) 2024-04-23T21:22:11.793 INFO:teuthology.orchestra.run.smithi111.stdout:4153 write 0x5bca506 thru 0x5bd05ea (0x60e5 bytes) 2024-04-23T21:22:11.796 INFO:teuthology.orchestra.run.smithi111.stdout:4155 write 0x7902ccb thru 0x790922c (0x6562 bytes) 2024-04-23T21:22:11.800 INFO:teuthology.orchestra.run.smithi111.stdout:4156 read 0x415b209 thru 0x41672c1 (0xc0b9 bytes) 2024-04-23T21:22:11.801 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:11.801 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:11.801 INFO:teuthology.orchestra.run.smithi111.stdout:4159 write 0x534662a thru 0x534871a (0x20f1 bytes) 2024-04-23T21:22:11.803 INFO:teuthology.orchestra.run.smithi111.stdout:4160 write 0x113fc00 thru 0x114d548 (0xd949 bytes) 2024-04-23T21:22:11.804 INFO:teuthology.orchestra.run.smithi111.stdout:4161 writesame 0x5625f9f thru 0x562a8e8 data_size 0x3b(0x494a bytes) 2024-04-23T21:22:11.817 INFO:teuthology.orchestra.run.smithi111.stdout:4162 writesame 0xdd18718 thru 0xdd1aa21 data_size 0x45(0x230a bytes) 2024-04-23T21:22:11.818 INFO:teuthology.orchestra.run.smithi111.stdout:4163 write 0x97945bf thru 0x979e23b (0x9c7d bytes) 2024-04-23T21:22:11.820 INFO:teuthology.orchestra.run.smithi111.stdout:4169 write 0x9a1f972 thru 0x9a22aaf (0x313e bytes) 2024-04-23T21:22:11.821 INFO:teuthology.orchestra.run.smithi111.stdout:4171 read 0x67dd5e8 thru 0x67e1977 (0x4390 bytes) 2024-04-23T21:22:11.822 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:11.822 INFO:teuthology.orchestra.run.smithi111.stdout:4173 write 0x5df7f01 thru 0x5dff54a (0x764a bytes) 2024-04-23T21:22:11.823 INFO:teuthology.orchestra.run.smithi111.stdout:4174 read 0x203cbe1 thru 0x204bbb5 (0xefd5 bytes) 2024-04-23T21:22:11.823 INFO:teuthology.orchestra.run.smithi111.stdout:4176 write 0x6611f96 thru 0x66147c0 (0x282b bytes) 2024-04-23T21:22:11.827 INFO:teuthology.orchestra.run.smithi111.stdout:4178 read 0x8d984e6 thru 0x8d9f6ce (0x71e9 bytes) 2024-04-23T21:22:11.827 INFO:teuthology.orchestra.run.smithi111.stdout:4179 write 0x91632ce thru 0x916777b (0x44ae bytes) 2024-04-23T21:22:11.828 INFO:teuthology.orchestra.run.smithi111.stdout:4180 punch from 0x3638bc8 to 0x3640db1, (0x81e9 bytes) 2024-04-23T21:22:11.828 INFO:teuthology.orchestra.run.smithi111.stdout:4181 write 0x3bd113f thru 0x3be0d5a (0xfc1c bytes) 2024-04-23T21:22:11.832 INFO:teuthology.orchestra.run.smithi111.stdout:4182 read 0xc9c33db thru 0xc9ccdf1 (0x9a17 bytes) 2024-04-23T21:22:11.833 INFO:teuthology.orchestra.run.smithi111.stdout:4183 read 0x23896a1 thru 0x238d099 (0x39f9 bytes) 2024-04-23T21:22:11.835 INFO:teuthology.orchestra.run.smithi111.stdout:4184 trunc from 0xdd1aa22 to 0xba0f647 2024-04-23T21:22:11.843 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:11.843 INFO:teuthology.orchestra.run.smithi111.stdout:4186 read 0xa4a8d2 thru 0xa57369 (0xca98 bytes) 2024-04-23T21:22:11.843 INFO:teuthology.orchestra.run.smithi111.stdout:4187 punch from 0x7bc9c5a to 0x7bd069b, (0x6a41 bytes) 2024-04-23T21:22:11.843 INFO:teuthology.orchestra.run.smithi111.stdout:4188 read 0x71d78ba thru 0x71e448e (0xcbd5 bytes) 2024-04-23T21:22:11.844 INFO:teuthology.orchestra.run.smithi111.stdout:4189 punch from 0x4590e21 to 0x459e5b6, (0xd795 bytes) 2024-04-23T21:22:11.845 INFO:teuthology.orchestra.run.smithi111.stdout:4192 punch from 0x85f281e to 0x85fd11a, (0xa8fc bytes) 2024-04-23T21:22:11.845 INFO:teuthology.orchestra.run.smithi111.stdout:4193 read 0x49c46cd thru 0x49d04c9 (0xbdfd bytes) 2024-04-23T21:22:11.845 INFO:teuthology.orchestra.run.smithi111.stdout:4196 writesame 0xc17638 thru 0xc20d5e data_size 0x5(0x9727 bytes) 2024-04-23T21:22:11.848 INFO:teuthology.orchestra.run.smithi111.stdout:4198 punch from 0x1e77dfd to 0x1e877ad, (0xf9b0 bytes) 2024-04-23T21:22:11.848 INFO:teuthology.orchestra.run.smithi111.stdout:4199 writesame 0x1578f61 thru 0x1587f7c data_size 0x7f(0xf01c bytes) 2024-04-23T21:22:11.849 INFO:teuthology.orchestra.run.smithi111.stdout:4200 clone 35 order 18 su 131072 sc 3 2024-04-23T21:22:11.855 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:11.855+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:11.858 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:11.858 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:11.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:11.956 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:11.956+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:12.009 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.009 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:12.059 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:12.059+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:12.158 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:12.159+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:12.160 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.160 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:12.259 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:12.260+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:12.293 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone34 from 0xba0f647 (overlap 0x47632ec) to 0x0 2024-04-23T21:22:12.325 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.326 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.335 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:12.353 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-04-23T21:22:12.375 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:12.376+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:12.462 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:12.463+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:12.479 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.480 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:12.563 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:12.563+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:12.608 INFO:teuthology.orchestra.run.smithi111.stdout:4201 writesame 0x2f33eab thru 0x2f34318 data_size 0x15(0x46e bytes) 2024-04-23T21:22:12.615 INFO:teuthology.orchestra.run.smithi111.stdout:4202 write 0x223f05 thru 0x231377 (0xd473 bytes) 2024-04-23T21:22:12.620 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.620 INFO:teuthology.orchestra.run.smithi111.stdout:4205 writesame 0x1adab36 thru 0x1ae61c7 data_size 0x1(0xb692 bytes) 2024-04-23T21:22:12.640 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.641 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.641 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.649 INFO:teuthology.orchestra.run.smithi111.stdout:4207 write 0xe555c7a thru 0xe561d8a (0xc111 bytes) 2024-04-23T21:22:12.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:12.653 INFO:teuthology.orchestra.run.smithi111.stdout:4209 writesame 0x9112cb6 thru 0x911e3f0 data_size 0x1(0xb73b bytes) 2024-04-23T21:22:12.664 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:12.664+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:12.677 INFO:teuthology.orchestra.run.smithi111.stdout:4211 read 0x4ba2a74 thru 0x4bb1504 (0xea91 bytes) 2024-04-23T21:22:12.677 INFO:teuthology.orchestra.run.smithi111.stdout:4212 write 0xcc89cc1 thru 0xcc8afc1 (0x1301 bytes) 2024-04-23T21:22:12.680 INFO:teuthology.orchestra.run.smithi111.stdout:4213 read 0x978a819 thru 0x978ca85 (0x226d bytes) 2024-04-23T21:22:12.680 INFO:teuthology.orchestra.run.smithi111.stdout:4214 writesame 0xaaf2823 thru 0xaaf47a7 data_size 0x1(0x1f85 bytes) 2024-04-23T21:22:12.687 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.687 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.687 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.687 INFO:teuthology.orchestra.run.smithi111.stdout:4219 write 0xc527a thru 0xcdfd2 (0x8d59 bytes) 2024-04-23T21:22:12.699 INFO:teuthology.orchestra.run.smithi111.stdout:4220 trunc from 0xe561d8b to 0x749dd69 2024-04-23T21:22:12.721 INFO:teuthology.orchestra.run.smithi111.stdout:4221 trunc from 0x749dd69 to 0x8795eb2 2024-04-23T21:22:12.739 INFO:teuthology.orchestra.run.smithi111.stdout:4222 writesame 0xbec12b9 thru 0xbec6802 data_size 0x3(0x554a bytes) 2024-04-23T21:22:12.739 INFO:teuthology.orchestra.run.smithi111.stdout:4224 write 0x7b65381 thru 0x7b6bb02 (0x6782 bytes) 2024-04-23T21:22:12.743 INFO:teuthology.orchestra.run.smithi111.stdout:4225 write 0xa8928d thru 0xa8b3f6 (0x216a bytes) 2024-04-23T21:22:12.747 INFO:teuthology.orchestra.run.smithi111.stdout:4226 trunc from 0xbec6803 to 0xad88048 2024-04-23T21:22:12.758 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.758 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.763 INFO:teuthology.orchestra.run.smithi111.stdout:4227 write 0xd0c8ff2 thru 0xd0cd1a4 (0x41b3 bytes) 2024-04-23T21:22:12.764 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:12.765+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:12.767 INFO:teuthology.orchestra.run.smithi111.stdout:4228 write 0xaee9cda thru 0xaeefa47 (0x5d6e bytes) 2024-04-23T21:22:12.768 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:12.769 INFO:teuthology.orchestra.run.smithi111.stdout:4229 read 0x6500543 thru 0x650f965 (0xf423 bytes) 2024-04-23T21:22:12.770 INFO:teuthology.orchestra.run.smithi111.stdout:4230 write 0x2cafd72 thru 0x2cb8dd3 (0x9062 bytes) 2024-04-23T21:22:12.773 INFO:teuthology.orchestra.run.smithi111.stdout:4231 read 0x6743133 thru 0x674db9f (0xaa6d bytes) 2024-04-23T21:22:12.774 INFO:teuthology.orchestra.run.smithi111.stdout:4232 write 0xd14463 thru 0xd14a00 (0x59e bytes) 2024-04-23T21:22:12.777 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:12.777 INFO:teuthology.orchestra.run.smithi111.stdout:4234 punch from 0x50cfbe8 to 0x50d1d3d, (0x2155 bytes) 2024-04-23T21:22:12.777 INFO:teuthology.orchestra.run.smithi111.stdout:4235 write 0x93292c3 thru 0x932fa34 (0x6772 bytes) 2024-04-23T21:22:12.779 INFO:teuthology.orchestra.run.smithi111.stdout:4236 read 0x4f43519 thru 0x4f4fd4a (0xc832 bytes) 2024-04-23T21:22:12.786 INFO:teuthology.orchestra.run.smithi111.stdout:4237 write 0xe367ed6 thru 0xe372008 (0xa133 bytes) 2024-04-23T21:22:12.789 INFO:teuthology.orchestra.run.smithi111.stdout:4238 writesame 0x99a5f2a thru 0x99a9cfe data_size 0x1(0x3dd5 bytes) 2024-04-23T21:22:12.798 INFO:teuthology.orchestra.run.smithi111.stdout:4239 punch from 0x9fb8961 to 0x9fc4fe3, (0xc682 bytes) 2024-04-23T21:22:12.802 INFO:teuthology.orchestra.run.smithi111.stdout:4240 trunc from 0xe372009 to 0xba22aa7 2024-04-23T21:22:12.812 INFO:teuthology.orchestra.run.smithi111.stdout:4241 write 0xaa2ac9f thru 0xaa33e1f (0x9181 bytes) 2024-04-23T21:22:12.815 INFO:teuthology.orchestra.run.smithi111.stdout:4242 punch from 0x56de48a to 0x56e51c2, (0x6d38 bytes) 2024-04-23T21:22:12.818 INFO:teuthology.orchestra.run.smithi111.stdout:4243 trunc from 0xba22aa7 to 0x91a4a10 2024-04-23T21:22:12.827 INFO:teuthology.orchestra.run.smithi111.stdout:4245 read 0x4c2751c thru 0x4c2c7f7 (0x52dc bytes) 2024-04-23T21:22:12.828 INFO:teuthology.orchestra.run.smithi111.stdout:4246 read 0x85a32dd thru 0x85a49e2 (0x1706 bytes) 2024-04-23T21:22:12.828 INFO:teuthology.orchestra.run.smithi111.stdout:4247 clone 36 order 23 su 65536 sc 10 2024-04-23T21:22:12.865 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:12.865+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:12.912 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:12.913 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:12.921 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:12.966 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:12.967+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:13.024 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:13.024 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:13.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:13.067 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:13.067+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:13.135 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:13.135 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:13.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:13.168 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:13.168+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:13.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:22:13.251 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:13.251 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:13.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:13.268 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:13.269+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:13.304 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone35 from 0x91a4a10 (overlap 0x749dd69) to 0x22fc26c 2024-04-23T21:22:13.370 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:13.370+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:13.439 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:13.440 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:13.452 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:13.493 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:13.493+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:13.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:13.608 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:13.608+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:13.614 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-04-23T21:22:13.624 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:13.624 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:13.626 INFO:teuthology.orchestra.run.smithi111.stdout:4249 write 0xa23c57c thru 0xa24073b (0x41c0 bytes) 2024-04-23T21:22:13.631 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:13.632 INFO:teuthology.orchestra.run.smithi111.stdout:4251 read 0x959b5a0 thru 0x95a95a9 (0xe00a bytes) 2024-04-23T21:22:13.632 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:13.632 INFO:teuthology.orchestra.run.smithi111.stdout:4253 write 0x7cbb003 thru 0x7cc4086 (0x9084 bytes) 2024-04-23T21:22:13.637 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:13.653 INFO:teuthology.orchestra.run.smithi111.stdout:4256 write 0x3013588 thru 0x302250d (0xef86 bytes) 2024-04-23T21:22:13.672 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:13.672+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:13.706 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:13.706 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:13.711 INFO:teuthology.orchestra.run.smithi111.stdout:4257 writesame 0x691ebdc thru 0x6923f9b data_size 0x20(0x53c0 bytes) 2024-04-23T21:22:13.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:13.727 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:13.737 INFO:teuthology.orchestra.run.smithi111.stdout:4258 punch from 0x1c89e2a to 0x1c8db9c, (0x3d72 bytes) 2024-04-23T21:22:13.738 INFO:teuthology.orchestra.run.smithi111.stdout:4260 read 0x9b64a86 thru 0x9b6fb59 (0xb0d4 bytes) 2024-04-23T21:22:13.738 INFO:teuthology.orchestra.run.smithi111.stdout:4262 write 0x24db522 thru 0x24dfdb5 (0x4894 bytes) 2024-04-23T21:22:13.746 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:13.746 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:13.753 INFO:teuthology.orchestra.run.smithi111.stdout:4263 trunc from 0xa24073c to 0xa46545 2024-04-23T21:22:13.772 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:13.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:13.772 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:13.774 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:13.773+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:13.788 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.847188806999998, 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53205 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 2932736~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002560248000002, 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013002390000000001, 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.793 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53205 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785413+0000", 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785414+0000", 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7599999999999997e-07 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785417+0000", 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0019999999999998e-06 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785427+0000", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5929999999999995e-06 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785441+0000", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4764e-05 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798225+0000", 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012784007 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798310+0000", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4618999999999997e-05 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798416+0000", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010582900000000001 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53206 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 3309568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:13.797 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002523647, 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013168311, 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53206 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785450+0000", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785451+0000", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785453+0000", 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4480000000000001e-06 2024-04-23T21:22:13.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785460+0000", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.742e-06 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785464+0000", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5390000000000001e-06 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798454+0000", 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012989908999999999 2024-04-23T21:22:13.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798540+0000", 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6206999999999996e-05 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798618+0000", 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7956999999999999e-05 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.801 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53258 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 753664~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002260482000001, 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013389243, 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53258 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785713+0000", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785714+0000", 2024-04-23T21:22:13.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6400000000000001e-07 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785724+0000", 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.8560000000000002e-06 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785736+0000", 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2339e-05 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785744+0000", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8050000000000007e-06 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798657+0000", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012913345999999999 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.798736+0000", 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8627000000000005e-05 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799102+0000", 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000366406 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53259 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 933888~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002217747, 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013579291, 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.806 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53259 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785756+0000", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785757+0000", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785759+0000", 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.481e-06 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785769+0000", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0234000000000001e-05 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785775+0000", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.648e-06 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799145+0000", 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013370253 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799229+0000", 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3820999999999996e-05 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799335+0000", 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000106226 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.810 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53260 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1114112~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002190582000001, 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01398165, 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53260 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785783+0000", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785784+0000", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.27e-07 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785786+0000", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2520000000000002e-06 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785792+0000", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6510000000000004e-06 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785798+0000", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3170000000000001e-06 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799366+0000", 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013568573 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799453+0000", 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6549000000000001e-05 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799765+0000", 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031188100000000002 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53261 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1294336~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002169264999999, 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014180051000000001, 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53261 2024-04-23T21:22:13.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785804+0000", 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785805+0000", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785807+0000", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0190000000000001e-06 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785814+0000", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785820+0000", 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1279999999999996e-06 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799793+0000", 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013973167 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799884+0000", 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1049000000000002e-05 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.799984+0000", 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010002500000000001 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53262 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1671168~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002146048, 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014362929, 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.819 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53262 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:13.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785828+0000", 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785830+0000", 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.131e-06 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785837+0000", 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.849e-06 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785842+0000", 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3859999999999998e-06 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800015+0000", 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014173008000000001 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800098+0000", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2268999999999994e-05 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800191+0000", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2818000000000005e-05 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53263 3.0 3:039b38a3:::rbd_data.11183e0a89c0.000000000000000b:16 [read 1851392~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:13.823 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002124224999999, 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014770998, 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53263 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785849+0000", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785850+0000", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6199999999999998e-07 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785852+0000", 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.046e-06 2024-04-23T21:22:13.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785858+0000", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4799999999999998e-06 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785864+0000", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1279999999999999e-06 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800236+0000", 2024-04-23T21:22:13.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014372433 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800327+0000", 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0816000000000006e-05 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800620+0000", 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00029363300000000001 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53264 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002103797, 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014801979999999999, 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53264 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785870+0000", 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:22:13.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785873+0000", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9160000000000001e-06 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785880+0000", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6599999999999998e-06 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785885+0000", 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.113e-06 2024-04-23T21:22:13.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800660+0000", 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775352 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800672+0000", 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1528e-05 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53265 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002071704999999, 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014804595, 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53265 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785902+0000", 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785905+0000", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4250000000000001e-06 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785912+0000", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3889999999999999e-06 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785924+0000", 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1953999999999999e-05 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800700+0000", 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014775307 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800707+0000", 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0489999999999998e-06 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53266 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.835 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.002022828000001, 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014787503, 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53266 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:13.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785951+0000", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9999999999999997e-07 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785954+0000", 2024-04-23T21:22:13.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.649e-06 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785969+0000", 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5058999999999999e-05 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785975+0000", 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9529999999999996e-06 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800730+0000", 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014755111 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800738+0000", 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1310000000000006e-06 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53267 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.001989865999999, 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014785247, 2024-04-23T21:22:13.839 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53267 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785984+0000", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2e-07 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785992+0000", 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8059999999999995e-06 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.785999+0000", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3520000000000001e-06 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786006+0000", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5549999999999999e-06 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800760+0000", 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014754259 2024-04-23T21:22:13.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800769+0000", 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.755e-06 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53268 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.001953618999998, 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014799623, 2024-04-23T21:22:13.843 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53268 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786020+0000", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786021+0000", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786023+0000", 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5569999999999998e-06 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786031+0000", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.644e-06 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786036+0000", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4890000000000003e-06 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800808+0000", 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014771467 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800820+0000", 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1924e-05 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:53269 3.0 3.c51cd9c0 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 22.001923881, 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014798240000000001, 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:13.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 53269 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786050+0000", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786057+0000", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3040000000000001e-06 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786065+0000", 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8640000000000002e-06 2024-04-23T21:22:13.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.786077+0000", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1595e-05 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800840+0000", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.014762542999999999 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:51.800848+0000", 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4249999999999998e-06 2024-04-23T21:22:13.850 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:61142 3.4 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 13.409820151, 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.019170651, 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.851 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 61142 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378154+0000", 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.378156+0000", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2989999999999999e-06 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383079+0000", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0049228930000000002 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383111+0000", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2338000000000001e-05 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383125+0000", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4039999999999999e-05 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383226+0000", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010133599999999999 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.383338+0000", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000111777 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:13.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384242+0000", 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00090330900000000003 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.395145+0000", 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010903042999999999 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397245+0000", 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002099952 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397289+0000", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4792000000000001e-05 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397324+0000", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4872000000000001e-05 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63700 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:13.856 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.6187833180000002, 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027843201000000001, 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63700 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:13.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169190+0000", 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169192+0000", 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5069999999999999e-06 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169200+0000", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1429999999999998e-06 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169214+0000", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4422e-05 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169221+0000", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6699999999999997e-06 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169280+0000", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.906e-05 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.169343+0000", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2478999999999996e-05 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.174410+0000", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.005067078 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:13.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.195126+0000", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.020716472 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196931+0000", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0018052929999999999 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.196984+0000", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2627999999999998e-05 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.197034+0000", 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9449e-05 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63757 3.19 3:9ccb22ce:::rbd_data.1118139b0f86.0000000000000025:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.5004574139999995, 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.015960412, 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:13.862 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63757 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287516+0000", 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287517+0000", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1900000000000001e-07 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287522+0000", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287533+0000", 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.079e-05 2024-04-23T21:22:13.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287540+0000", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2019999999999998e-06 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287617+0000", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7014000000000005e-05 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.287667+0000", 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9421e-05 2024-04-23T21:22:13.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 3", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.290130+0000", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0024636150000000002 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303396+0000", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013265476 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303410+0000", 2024-04-23T21:22:13.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3946e-05 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303446+0000", 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.612e-05 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.303477+0000", 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0839000000000001e-05 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:63940 3.4 3:217d9948:::rbd_data.1118139b0f86.0000000000000056:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 8.3153568579999995, 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.016049744000000001, 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 63940 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472617+0000", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472618+0000", 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.61e-07 2024-04-23T21:22:13.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472622+0000", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1690000000000002e-06 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472632+0000", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0399e-05 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472637+0000", 2024-04-23T21:22:13.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2310000000000004e-06 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472674+0000", 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6465999999999999e-05 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.472718+0000", 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580999999999999e-05 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:13.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.475549+0000", 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0028307340000000001 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488593+0000", 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013043406 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488604+0000", 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1649e-05 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488636+0000", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.1622999999999999e-05 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.488667+0000", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0725000000000001e-05 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:64217 3.6 3:67a8ca51:::rbd_data.1118139b0f86.0000000000000049:head [set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:13.873 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 7.9714182989999998, 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.013303083, 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 64217 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:13.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816555+0000", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816556+0000", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2500000000000005e-07 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816560+0000", 2024-04-23T21:22:13.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2030000000000002e-06 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816571+0000", 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0514e-05 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816576+0000", 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9169999999999997e-06 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:13.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816618+0000", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2190000000000001e-05 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.816710+0000", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2008000000000004e-05 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.818915+0000", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.002205046 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:13.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829742+0000", 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010826987 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829794+0000", 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2126000000000002e-05 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829828+0000", 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4406999999999998e-05 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:13.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:05.829858+0000", 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9960000000000001e-05 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:13.879 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:13.880 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:13.881+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:13.881 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:13.881 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:13.882 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:13.882 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:13.882 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:13.891 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:13.892 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:13.914 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:13.915 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:13.917 INFO:teuthology.orchestra.run.smithi111.stdout:4265 writesame 0x69cc105 thru 0x69d6474 data_size 0x28(0xa370 bytes) 2024-04-23T21:22:13.923 INFO:teuthology.orchestra.run.smithi111.stdout:4268 trunc from 0x69d6475 to 0x260b56f 2024-04-23T21:22:13.946 INFO:teuthology.orchestra.run.smithi111.stdout:4271 writesame 0xa628a7e thru 0xa6371af data_size 0x65(0xe732 bytes) 2024-04-23T21:22:13.946 INFO:teuthology.orchestra.run.smithi111.stdout:4273 writesame 0x356e8f2 thru 0x3575046 data_size 0x1(0x6755 bytes) 2024-04-23T21:22:13.948 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:13.948 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:13.958 INFO:teuthology.orchestra.run.smithi111.stdout:4274 read 0x79ccce4 thru 0x79cf1c2 (0x24df bytes) 2024-04-23T21:22:13.958 INFO:teuthology.orchestra.run.smithi111.stdout:4276 read 0x58cca50 thru 0x58d3780 (0x6d31 bytes) 2024-04-23T21:22:13.959 INFO:teuthology.orchestra.run.smithi111.stdout:4277 trunc from 0xa6371b0 to 0xd2c8460 2024-04-23T21:22:13.964 INFO:teuthology.orchestra.run.smithi111.stdout:4281 read 0xbccfec thru 0xbda5cc (0xd5e1 bytes) 2024-04-23T21:22:13.975 INFO:teuthology.orchestra.run.smithi111.stdout:4282 trunc from 0xd2c8460 to 0x4d970f3 2024-04-23T21:22:13.981 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:13.982+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:13.999 INFO:teuthology.orchestra.run.smithi111.stdout:4283 write 0xc521a7f thru 0xc52bca0 (0xa222 bytes) 2024-04-23T21:22:14.002 INFO:teuthology.orchestra.run.smithi111.stdout:4284 write 0x71d7553 thru 0x71e6bd4 (0xf682 bytes) 2024-04-23T21:22:14.007 INFO:teuthology.orchestra.run.smithi111.stdout:4287 write 0x960de44 thru 0x961a48f (0xc64c bytes) 2024-04-23T21:22:14.013 INFO:teuthology.orchestra.run.smithi111.stdout:4288 write 0x21648b thru 0x225ac7 (0xf63d bytes) 2024-04-23T21:22:14.015 INFO:teuthology.orchestra.run.smithi111.stdout:4290 writesame 0xc03210a thru 0xc03fc52 data_size 0x1(0xdb49 bytes) 2024-04-23T21:22:14.018 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.018 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:14.018 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:14.018 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.027 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:14.043 INFO:teuthology.orchestra.run.smithi111.stdout:4292 read 0xb1ed50a thru 0xb1f4c73 (0x776a bytes) 2024-04-23T21:22:14.050 INFO:teuthology.orchestra.run.smithi111.stdout:4293 write 0xc56cca8 thru 0xc579cd2 (0xd02b bytes) 2024-04-23T21:22:14.054 INFO:teuthology.orchestra.run.smithi111.stdout:4294 trunc from 0xc579cd3 to 0xc34fd57 2024-04-23T21:22:14.063 INFO:teuthology.orchestra.run.smithi111.stdout:4295 read 0x73ccbc1 thru 0x73db95d (0xed9d bytes) 2024-04-23T21:22:14.064 INFO:teuthology.orchestra.run.smithi111.stdout:4296 write 0x63cfd06 thru 0x63dfbbb (0xfeb6 bytes) 2024-04-23T21:22:14.068 INFO:teuthology.orchestra.run.smithi111.stdout:4297 write 0x2570fed thru 0x2579eb9 (0x8ecd bytes) 2024-04-23T21:22:14.070 INFO:teuthology.orchestra.run.smithi111.stdout:4298 writesame 0xaf46dce thru 0xaf4d096 data_size 0x1(0x62c9 bytes) 2024-04-23T21:22:14.080 INFO:teuthology.orchestra.run.smithi111.stdout:4299 write 0x2228b7c thru 0x2229967 (0xdec bytes) 2024-04-23T21:22:14.082 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:14.082+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:14.092 INFO:teuthology.orchestra.run.smithi111.stdout:4300 write 0xdeb2f6f thru 0xdeb750c (0x459e bytes) 2024-04-23T21:22:14.095 INFO:teuthology.orchestra.run.smithi111.stdout:4301 write 0x6fa33fa thru 0x6fa5805 (0x240c bytes) 2024-04-23T21:22:14.097 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.097 INFO:teuthology.orchestra.run.smithi111.stdout:4305 punch from 0xabce3b9 to 0xabdc30b, (0xdf52 bytes) 2024-04-23T21:22:14.097 INFO:teuthology.orchestra.run.smithi111.stdout:4306 punch from 0xcb15276 to 0xcb18370, (0x30fa bytes) 2024-04-23T21:22:14.113 INFO:teuthology.orchestra.run.smithi111.stdout:4307 trunc from 0xdeb750d to 0x3c0c2c 2024-04-23T21:22:14.125 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:14.125 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:14.140 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-23T21:22:13.603917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1660088752,"num_objects":1153,"num_object_clones":239,"num_object_copies":2306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":448,"num_read":91461,"num_read_kb":617035,"num_write":12030,"num_write_kb":371792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9515,"ondisk_log_size":9515,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":354564,"kb_used_data":60776,"kb_used_omap":17,"kb_used_meta":293678,"kb_avail":754620156,"statfs":{"total":773094113280,"available":772731039744,"internally_reserved":0,"allocated":62234624,"data_stored":517829025,"data_compressed":30452181,"data_compressed_allocated":34451456,"data_compressed_original":496593460,"omap_allocated":17904,"internal_metadata":300726800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29234174,"num_objects":14,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":190,"num_read_kb":4603,"num_write":68,"num_write_kb":2703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000826"},"pg_stats":[{"pgid":"3.1f","version":"64'276","reported_seq":2575,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.350420+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:22:08.350420+0000","last_peered":"2024-04-23T21:22:08.350420+0000","last_clean":"2024-04-23T21:22:08.350420+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:08.350420+0000","last_undegraded":"2024-04-23T21:22:08.350420+0000","last_fullsized":"2024-04-23T21:22:08.350420+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":276,"log_dups_size":0,"ondisk_log_size":276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:28:04.714028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":68500321,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":13,"num_read":2707,"num_read_kb":12451,"num_write":346,"num_write_kb":26545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"64'263","reported_seq":3487,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.352959+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:22:08.352959+0000","last_peered":"2024-04-23T21:22:08.352959+0000","last_clean":"2024-04-23T21:22:08.352959+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:22:08.352959+0000","last_undegraded":"2024-04-23T21:22:08.352959+0000","last_fullsized":"2024-04-23T21:22:08.352959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:49:43.479235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":55274288,"num_objects":41,"num_object_clones":10,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":17,"num_read":3508,"num_read_kb":33007,"num_write":303,"num_write_kb":16144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'185","reported_seq":3194,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.213588+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:22:08.213588+0000","last_peered":"2024-04-23T21:22:08.213588+0000","last_clean":"2024-04-23T21:22:08.213588+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:08.213588+0000","last_undegraded":"2024-04-23T21:22:08.213588+0000","last_fullsized":"2024-04-23T21:22:08.213588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:54:29.678682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":22295959,"num_objects":35,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":18,"num_read":3417,"num_read_kb":11870,"num_write":172,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"64'273","reported_seq":1916,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.359505+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:22:08.359505+0000","last_peered":"2024-04-23T21:22:08.359505+0000","last_clean":"2024-04-23T21:22:08.359505+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:08.359505+0000","last_undegraded":"2024-04-23T21:22:08.359505+0000","last_fullsized":"2024-04-23T21:22:08.359505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:12:15.447829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":64577584,"num_objects":48,"num_object_clones":13,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":19,"num_read":1949,"num_read_kb":14806,"num_write":335,"num_write_kb":11274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"60'212","reported_seq":6361,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.216200+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:22:08.216200+0000","last_peered":"2024-04-23T21:22:08.216200+0000","last_clean":"2024-04-23T21:22:08.216200+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:08.216200+0000","last_undegraded":"2024-04-23T21:22:08.216200+0000","last_fullsized":"2024-04-23T21:22:08.216200+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:29:44.197286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":91250187,"num_objects":39,"num_object_clones":12,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":11,"num_read":6597,"num_read_kb":37202,"num_write":248,"num_write_kb":51140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"64'556","reported_seq":3282,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.350383+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:22:08.350383+0000","last_peered":"2024-04-23T21:22:08.350383+0000","last_clean":"2024-04-23T21:22:08.350383+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:08.350383+0000","last_undegraded":"2024-04-23T21:22:08.350383+0000","last_fullsized":"2024-04-23T21:22:08.350383+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":556,"log_dups_size":0,"ondisk_log_size":556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:36:54.655159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":45686038,"num_objects":38,"num_object_clones":8,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":15,"num_read":5739,"num_read_kb":36031,"num_write":768,"num_write_kb":23093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'357","reported_seq":1363,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-23T21:22:12.715737+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:22:12.715737+0000","last_peered":"2024-04-23T21:22:12.715737+0000","last_clean":"2024-04-23T21:22:12.715737+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:22:12.715737+0000","last_undegraded":"2024-04-23T21:22:12.715737+0000","last_fullsized":"2024-04-23T21:22:12.715737+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:31:27.391159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":38400005,"num_objects":38,"num_object_clones":8,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":18,"num_read":2370,"num_read_kb":11447,"num_write":498,"num_write_kb":5202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"64'546","reported_seq":2790,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.368283+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:22:08.368283+0000","last_peered":"2024-04-23T21:22:08.368283+0000","last_clean":"2024-04-23T21:22:08.368283+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:22:08.368283+0000","last_undegraded":"2024-04-23T21:22:08.368283+0000","last_fullsized":"2024-04-23T21:22:08.368283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":546,"log_dups_size":0,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:17.500689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":33767962,"num_objects":46,"num_object_clones":12,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":21,"num_read":5122,"num_read_kb":10884,"num_write":729,"num_write_kb":2403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"64'221","reported_seq":1583,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.365897+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:22:08.365897+0000","last_peered":"2024-04-23T21:22:08.365897+0000","last_clean":"2024-04-23T21:22:08.365897+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:08.365897+0000","last_undegraded":"2024-04-23T21:22:08.365897+0000","last_fullsized":"2024-04-23T21:22:08.365897+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:10:00.676519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":32059570,"num_objects":40,"num_object_clones":5,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":19,"num_read":2382,"num_read_kb":25807,"num_write":222,"num_write_kb":14174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"64'153","reported_seq":1016,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.279640+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:22:08.279640+0000","last_peered":"2024-04-23T21:22:08.279640+0000","last_clean":"2024-04-23T21:22:08.279640+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:08.279640+0000","last_undegraded":"2024-04-23T21:22:08.279640+0000","last_fullsized":"2024-04-23T21:22:08.279640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:37.629771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":21341789,"num_objects":31,"num_object_clones":5,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":18,"num_read":763,"num_read_kb":9602,"num_write":151,"num_write_kb":2844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'246","reported_seq":3431,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.350385+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:22:08.350385+0000","last_peered":"2024-04-23T21:22:08.350385+0000","last_clean":"2024-04-23T21:22:08.350385+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:22:08.350385+0000","last_undegraded":"2024-04-23T21:22:08.350385+0000","last_fullsized":"2024-04-23T21:22:08.350385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:22:36.476623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":56043662,"num_objects":43,"num_object_clones":12,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":14,"num_read":3691,"num_read_kb":13334,"num_write":260,"num_write_kb":6196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"64'454","reported_seq":1685,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.353459+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:22:08.353459+0000","last_peered":"2024-04-23T21:22:08.353459+0000","last_clean":"2024-04-23T21:22:08.353459+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:08.353459+0000","last_undegraded":"2024-04-23T21:22:08.353459+0000","last_fullsized":"2024-04-23T21:22:08.353459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":454,"log_dups_size":0,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:35.891626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":88753165,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":15,"num_read":2844,"num_read_kb":28029,"num_write":609,"num_write_kb":9735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.228948+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:22:08.228948+0000","last_peered":"2024-04-23T21:22:08.228948+0000","last_clean":"2024-04-23T21:22:08.228948+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:22:08.228948+0000","last_undegraded":"2024-04-23T21:22:08.228948+0000","last_fullsized":"2024-04-23T21:22:08.228948+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"64'295","reported_seq":1200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.308178+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:22:08.308178+0000","last_peered":"2024-04-23T21:22:08.308178+0000","last_clean":"2024-04-23T21:22:08.308178+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:22:08.308178+0000","last_undegraded":"2024-04-23T21:22:08.308178+0000","last_fullsized":"2024-04-23T21:22:08.308178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:16:00.458069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":52321342,"num_objects":40,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":15,"num_read":1446,"num_read_kb":10906,"num_write":372,"num_write_kb":4654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211273+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:08.211273+0000","last_peered":"2024-04-23T21:22:08.211273+0000","last_clean":"2024-04-23T21:22:08.211273+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:08.211273+0000","last_undegraded":"2024-04-23T21:22:08.211273+0000","last_fullsized":"2024-04-23T21:22:08.211273+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"68'186","reported_seq":1264,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-23T21:22:12.301413+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:12.301413+0000","last_peered":"2024-04-23T21:22:12.301413+0000","last_clean":"2024-04-23T21:22:12.301413+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:12.301413+0000","last_undegraded":"2024-04-23T21:22:12.301413+0000","last_fullsized":"2024-04-23T21:22:12.301413+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:51:59.682308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":73442714,"num_objects":32,"num_object_clones":8,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":10,"num_read":1223,"num_read_kb":27550,"num_write":179,"num_write_kb":13101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.216279+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:08.216279+0000","last_peered":"2024-04-23T21:22:08.216279+0000","last_clean":"2024-04-23T21:22:08.216279+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:08.216279+0000","last_undegraded":"2024-04-23T21:22:08.216279+0000","last_fullsized":"2024-04-23T21:22:08.216279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'240","reported_seq":1159,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.220944+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:22:08.220944+0000","last_peered":"2024-04-23T21:22:08.220944+0000","last_clean":"2024-04-23T21:22:08.220944+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:22:08.220944+0000","last_undegraded":"2024-04-23T21:22:08.220944+0000","last_fullsized":"2024-04-23T21:22:08.220944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:31:05.105939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":38429773,"num_objects":28,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":11,"num_read":1396,"num_read_kb":7018,"num_write":331,"num_write_kb":2484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.220862+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:22:08.220862+0000","last_peered":"2024-04-23T21:22:08.220862+0000","last_clean":"2024-04-23T21:22:08.220862+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:22:08.220862+0000","last_undegraded":"2024-04-23T21:22:08.220862+0000","last_fullsized":"2024-04-23T21:22:08.220862+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"67'199","reported_seq":435,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-23T21:22:12.408615+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:12.408615+0000","last_peered":"2024-04-23T21:22:12.408615+0000","last_clean":"2024-04-23T21:22:12.408615+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:12.408615+0000","last_undegraded":"2024-04-23T21:22:12.408615+0000","last_fullsized":"2024-04-23T21:22:12.408615+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:28.627680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":67741909,"num_objects":45,"num_object_clones":10,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":20,"num_read":2462,"num_read_kb":18730,"num_write":223,"num_write_kb":12609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"64'168","reported_seq":883,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.350640+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:22:08.350640+0000","last_peered":"2024-04-23T21:22:08.350640+0000","last_clean":"2024-04-23T21:22:08.350640+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:08.350640+0000","last_undegraded":"2024-04-23T21:22:08.350640+0000","last_fullsized":"2024-04-23T21:22:08.350640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:11:54.107745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":108026902,"num_objects":36,"num_object_clones":7,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":13,"num_read":656,"num_read_kb":41333,"num_write":190,"num_write_kb":23639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":263,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211035+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:08.211035+0000","last_peered":"2024-04-23T21:22:08.211035+0000","last_clean":"2024-04-23T21:22:08.211035+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:08.211035+0000","last_undegraded":"2024-04-23T21:22:08.211035+0000","last_fullsized":"2024-04-23T21:22:08.211035+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":64,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-23T21:22:12.230667+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:12.230667+0000","last_peered":"2024-04-23T21:22:12.230667+0000","last_clean":"2024-04-23T21:22:12.230667+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:12.230667+0000","last_undegraded":"2024-04-23T21:22:12.230667+0000","last_fullsized":"2024-04-23T21:22:12.230667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"68'230","reported_seq":588,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-23T21:22:12.781340+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:12.781340+0000","last_peered":"2024-04-23T21:22:12.781340+0000","last_clean":"2024-04-23T21:22:12.781340+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:12.781340+0000","last_undegraded":"2024-04-23T21:22:12.781340+0000","last_fullsized":"2024-04-23T21:22:12.781340+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:02:34.583550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":17958446,"num_objects":34,"num_object_clones":6,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":17,"num_read":3109,"num_read_kb":7856,"num_write":284,"num_write_kb":1735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211079+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:08.211079+0000","last_peered":"2024-04-23T21:22:08.211079+0000","last_clean":"2024-04-23T21:22:08.211079+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:08.211079+0000","last_undegraded":"2024-04-23T21:22:08.211079+0000","last_fullsized":"2024-04-23T21:22:08.211079+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"64'536","reported_seq":4474,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.357728+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:22:08.357728+0000","last_peered":"2024-04-23T21:22:08.357728+0000","last_clean":"2024-04-23T21:22:08.357728+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:08.357728+0000","last_undegraded":"2024-04-23T21:22:08.357728+0000","last_fullsized":"2024-04-23T21:22:08.357728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":536,"log_dups_size":0,"ondisk_log_size":536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:09:58.005953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":74718534,"num_objects":49,"num_object_clones":13,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":16,"num_read":6841,"num_read_kb":40930,"num_write":813,"num_write_kb":27514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.213528+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:08.213528+0000","last_peered":"2024-04-23T21:22:08.213528+0000","last_clean":"2024-04-23T21:22:08.213528+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:08.213528+0000","last_undegraded":"2024-04-23T21:22:08.213528+0000","last_fullsized":"2024-04-23T21:22:08.213528+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"64'434","reported_seq":3561,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.352091+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:22:08.352091+0000","last_peered":"2024-04-23T21:22:08.352091+0000","last_clean":"2024-04-23T21:22:08.352091+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:22:08.352091+0000","last_undegraded":"2024-04-23T21:22:08.352091+0000","last_fullsized":"2024-04-23T21:22:08.352091+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:05:39.045725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":51347263,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":10,"num_read":5798,"num_read_kb":9235,"num_write":595,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":108,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211039+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:08.211039+0000","last_peered":"2024-04-23T21:22:08.211039+0000","last_clean":"2024-04-23T21:22:08.211039+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:08.211039+0000","last_undegraded":"2024-04-23T21:22:08.211039+0000","last_fullsized":"2024-04-23T21:22:08.211039+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"64'127","reported_seq":1132,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.283110+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:22:08.283110+0000","last_peered":"2024-04-23T21:22:08.283110+0000","last_clean":"2024-04-23T21:22:08.283110+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:22:08.283110+0000","last_undegraded":"2024-04-23T21:22:08.283110+0000","last_fullsized":"2024-04-23T21:22:08.283110+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:12:28.460672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":25554700,"num_objects":31,"num_object_clones":9,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":11,"num_read":1118,"num_read_kb":12301,"num_write":169,"num_write_kb":8576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"64'187","reported_seq":3306,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.240470+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:22:08.240470+0000","last_peered":"2024-04-23T21:22:08.240470+0000","last_clean":"2024-04-23T21:22:08.240470+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:08.240470+0000","last_undegraded":"2024-04-23T21:22:08.240470+0000","last_fullsized":"2024-04-23T21:22:08.240470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:22:00.483130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":63735324,"num_objects":43,"num_object_clones":12,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":19,"num_read":3121,"num_read_kb":27288,"num_write":215,"num_write_kb":10330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"64'351","reported_seq":2255,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.353668+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:22:08.353668+0000","last_peered":"2024-04-23T21:22:08.353668+0000","last_clean":"2024-04-23T21:22:08.353668+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:08.353668+0000","last_undegraded":"2024-04-23T21:22:08.353668+0000","last_fullsized":"2024-04-23T21:22:08.353668+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:28:17.796203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":19918020,"num_objects":34,"num_object_clones":5,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":15,"num_read":4170,"num_read_kb":9398,"num_write":350,"num_write_kb":10485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"64'260","reported_seq":1178,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.220801+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:22:08.220801+0000","last_peered":"2024-04-23T21:22:08.220801+0000","last_clean":"2024-04-23T21:22:08.220801+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:22:08.220801+0000","last_undegraded":"2024-04-23T21:22:08.220801+0000","last_fullsized":"2024-04-23T21:22:08.220801+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:00.624249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":43112736,"num_objects":36,"num_object_clones":7,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":13,"num_read":1412,"num_read_kb":22919,"num_write":281,"num_write_kb":9777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"64'206","reported_seq":810,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.350650+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:22:08.350650+0000","last_peered":"2024-04-23T21:22:08.350650+0000","last_clean":"2024-04-23T21:22:08.350650+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:22:08.350650+0000","last_undegraded":"2024-04-23T21:22:08.350650+0000","last_fullsized":"2024-04-23T21:22:08.350650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:32:58.533698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":66725508,"num_objects":31,"num_object_clones":9,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":11,"num_read":995,"num_read_kb":13762,"num_write":209,"num_write_kb":12528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"64'153","reported_seq":986,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211297+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:22:08.211297+0000","last_peered":"2024-04-23T21:22:08.211297+0000","last_clean":"2024-04-23T21:22:08.211297+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:08.211297+0000","last_undegraded":"2024-04-23T21:22:08.211297+0000","last_fullsized":"2024-04-23T21:22:08.211297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:20:45.835571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":26169058,"num_objects":28,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":11,"num_read":773,"num_read_kb":8753,"num_write":152,"num_write_kb":2259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"64'239","reported_seq":811,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.240394+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:22:08.240394+0000","last_peered":"2024-04-23T21:22:08.240394+0000","last_clean":"2024-04-23T21:22:08.240394+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:08.240394+0000","last_undegraded":"2024-04-23T21:22:08.240394+0000","last_fullsized":"2024-04-23T21:22:08.240394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:00:30.535930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":18136170,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":8,"num_read":1470,"num_read_kb":4040,"num_write":278,"num_write_kb":3059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'737","reported_seq":4890,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.376386+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:22:08.376386+0000","last_peered":"2024-04-23T21:22:08.376386+0000","last_clean":"2024-04-23T21:22:08.376386+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:08.376386+0000","last_undegraded":"2024-04-23T21:22:08.376386+0000","last_fullsized":"2024-04-23T21:22:08.376386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:36:07.387973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":56090883,"num_objects":37,"num_object_clones":4,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":11,"num_read":7824,"num_read_kb":29191,"num_write":1140,"num_write_kb":5719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"62'243","reported_seq":1170,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.228899+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:22:08.228899+0000","last_peered":"2024-04-23T21:22:08.228899+0000","last_clean":"2024-04-23T21:22:08.228899+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:08.228899+0000","last_undegraded":"2024-04-23T21:22:08.228899+0000","last_fullsized":"2024-04-23T21:22:08.228899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:46.920310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":78054160,"num_objects":29,"num_object_clones":3,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":8,"num_read":1427,"num_read_kb":17577,"num_write":309,"num_write_kb":8548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"64'156","reported_seq":811,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.349294+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:22:08.349294+0000","last_peered":"2024-04-23T21:22:08.349294+0000","last_clean":"2024-04-23T21:22:08.349294+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:08.349294+0000","last_undegraded":"2024-04-23T21:22:08.349294+0000","last_fullsized":"2024-04-23T21:22:08.349294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:00:42.588867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":46855140,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":8,"num_read":596,"num_read_kb":14824,"num_write":179,"num_write_kb":7135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"64'379","reported_seq":1226,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.211143+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:22:08.211143+0000","last_peered":"2024-04-23T21:22:08.211143+0000","last_clean":"2024-04-23T21:22:08.211143+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:08.211143+0000","last_undegraded":"2024-04-23T21:22:08.211143+0000","last_fullsized":"2024-04-23T21:22:08.211143+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:31.669318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":45041690,"num_objects":33,"num_object_clones":2,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":9,"num_read":2464,"num_read_kb":28039,"num_write":522,"num_write_kb":14062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"64'330","reported_seq":1180,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-23T21:22:08.359895+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:22:08.359895+0000","last_peered":"2024-04-23T21:22:08.359895+0000","last_clean":"2024-04-23T21:22:08.359895+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:08.359895+0000","last_undegraded":"2024-04-23T21:22:08.359895+0000","last_fullsized":"2024-04-23T21:22:08.359895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:44:07.944285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":68167582,"num_objects":40,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":14,"num_read":1955,"num_read_kb":20815,"num_write":439,"num_write_kb":16921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1659498384,"num_objects":1151,"num_object_clones":239,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":448,"num_read":91345,"num_read_kb":616935,"num_write":11871,"num_write_kb":369376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":56328192,"data_stored":514409559,"data_compressed":30523781,"data_compressed_allocated":34508800,"data_compressed_original":495913524,"omap_allocated":0,"internal_metadata":0},"log_size":9398,"ondisk_log_size":9398,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47852,"kb_used_data":8352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323988,"statfs":{"total":96636764160,"available":96587763712,"internally_reserved":0,"allocated":8552448,"data_stored":59105874,"data_compressed":4066534,"data_compressed_allocated":4677632,"data_compressed_original":56149732,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33828,"kb_used_data":2644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338012,"statfs":{"total":96636764160,"available":96602124288,"internally_reserved":0,"allocated":2707456,"data_stored":19436264,"data_compressed":972436,"data_compressed_allocated":1089536,"data_compressed_original":18392979,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":6060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":6205440,"data_stored":44053677,"data_compressed":2474541,"data_compressed_allocated":2998272,"data_compressed_original":41704134,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59364,"kb_used_data":14296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94312476,"statfs":{"total":96636764160,"available":96575975424,"internally_reserved":0,"allocated":14639104,"data_stored":150129405,"data_compressed":8908309,"data_compressed_allocated":9347072,"data_compressed_original":145877624,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47064,"kb_used_data":7568,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94324776,"statfs":{"total":96636764160,"available":96588570624,"internally_reserved":0,"allocated":7749632,"data_stored":47176460,"data_compressed":2651620,"data_compressed_allocated":3268608,"data_compressed_original":43634590,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34492,"kb_used_data":3312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337348,"statfs":{"total":96636764160,"available":96601444352,"internally_reserved":0,"allocated":3391488,"data_stored":20710073,"data_compressed":1178396,"data_compressed_allocated":1523712,"data_compressed_original":19433910,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39188,"kb_used_data":6664,"kb_used_omap":6,"kb_used_meta":32505,"kb_avail":94332652,"statfs":{"total":96636764160,"available":96596635648,"internally_reserved":0,"allocated":6823936,"data_stored":68402461,"data_compressed":4178671,"data_compressed_allocated":4059136,"data_compressed_original":66318010,"omap_allocated":6748,"internal_metadata":33285540},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51376,"kb_used_data":11880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94320464,"statfs":{"total":96636764160,"available":96584155136,"internally_reserved":0,"allocated":12165120,"data_stored":108814811,"data_compressed":6021674,"data_compressed_allocated":7487488,"data_compressed_original":105082481,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11452416,"data_stored":108605265,"data_compressed":6054099,"data_compressed_allocated":7524352,"data_compressed_original":105213553,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6008832,"data_stored":67462821,"data_compressed":4144613,"data_compressed_allocated":4014080,"data_compressed_original":65728186,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2658304,"data_stored":20375159,"data_compressed":1178396,"data_compressed_allocated":1523712,"data_compressed_original":19433910,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7049216,"data_stored":46947415,"data_compressed":2688235,"data_compressed_allocated":3305472,"data_compressed_original":43745182,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13893632,"data_stored":149782222,"data_compressed":8908309,"data_compressed_allocated":9347072,"data_compressed_original":145877624,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":43848731,"data_compressed":2506966,"data_compressed_allocated":3035136,"data_compressed_original":41835206,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1974272,"data_stored":19101350,"data_compressed":972436,"data_compressed_allocated":1089536,"data_compressed_original":18392979,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":7786496,"data_stored":58286596,"data_compressed":4070727,"data_compressed_allocated":4669440,"data_compressed_original":55686884,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:14.143 INFO:teuthology.orchestra.run.smithi111.stdout:4308 trunc from 0x3c0c2c to 0x8a234f6 2024-04-23T21:22:14.147 INFO:teuthology.orchestra.run.smithi111.stdout:4309 read 0x1625607 thru 0x1633941 (0xe33b bytes) 2024-04-23T21:22:14.148 INFO:teuthology.orchestra.run.smithi111.stdout:4310 writesame 0x6876926 thru 0x6882a06 data_size 0x3(0xc0e1 bytes) 2024-04-23T21:22:14.157 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.162 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.162 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:14.162 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:14.162 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:14.162 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.170 INFO:teuthology.orchestra.run.smithi111.stdout:4312 write 0xe350e29 thru 0xe353311 (0x24e9 bytes) 2024-04-23T21:22:14.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:14.174 INFO:teuthology.orchestra.run.smithi111.stdout:4314 writesame 0xb07d59a thru 0xb083467 data_size 0xf(0x5ece bytes) 2024-04-23T21:22:14.178 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.178 INFO:teuthology.orchestra.run.smithi111.stdout:4319 punch from 0xe0f813b to 0xe101f79, (0x9e3e bytes) 2024-04-23T21:22:14.178 INFO:teuthology.orchestra.run.smithi111.stdout:4320 read 0x682acf5 thru 0x682c532 (0x183e bytes) 2024-04-23T21:22:14.182 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:14.183+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:14.190 INFO:teuthology.orchestra.run.smithi111.stdout:4321 trunc from 0xe353312 to 0x40ded1a 2024-04-23T21:22:14.203 INFO:teuthology.orchestra.run.smithi111.stdout:4322 write 0x708658 thru 0x711096 (0x8a3f bytes) 2024-04-23T21:22:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:4323 punch from 0x51052 to 0x5c750, (0xb6fe bytes) 2024-04-23T21:22:14.204 INFO:teuthology.orchestra.run.smithi111.stdout:4324 write 0x11a0bad thru 0x11a6bc7 (0x601b bytes) 2024-04-23T21:22:14.219 INFO:teuthology.orchestra.run.smithi111.stdout:4325 write 0xafc891f thru 0xafcfe3d (0x751f bytes) 2024-04-23T21:22:14.230 INFO:teuthology.orchestra.run.smithi111.stdout:4326 writesame 0xd554e75 thru 0xd55f6e5 data_size 0x1(0xa871 bytes) 2024-04-23T21:22:14.230 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.230 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.230 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.231 INFO:teuthology.orchestra.run.smithi111.stdout:4334 read 0xcf32c87 thru 0xcf382d6 (0x5650 bytes) 2024-04-23T21:22:14.231 INFO:teuthology.orchestra.run.smithi111.stdout:4335 writesame 0x98c0b6a thru 0x98c6668 data_size 0x5(0x5aff bytes) 2024-04-23T21:22:14.234 INFO:teuthology.orchestra.run.smithi111.stdout:4336 read 0xae138d5 thru 0xae21479 (0xdba5 bytes) 2024-04-23T21:22:14.235 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.240 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:22:14.247 INFO:teuthology.orchestra.run.smithi111.stdout:4339 trunc from 0xd55f6e6 to 0x35f172b 2024-04-23T21:22:14.265 INFO:teuthology.orchestra.run.smithi111.stdout:4340 read 0x2dba170 thru 0x2dc2702 (0x8593 bytes) 2024-04-23T21:22:14.265 INFO:teuthology.orchestra.run.smithi111.stdout:4341 read 0x636548 thru 0x63b349 (0x4e02 bytes) 2024-04-23T21:22:14.266 INFO:teuthology.orchestra.run.smithi111.stdout:4342 punch from 0xb520b9 to 0xb5e502, (0xc449 bytes) 2024-04-23T21:22:14.266 INFO:teuthology.orchestra.run.smithi111.stdout:4344 punch from 0x7920 to 0x14470, (0xcb50 bytes) 2024-04-23T21:22:14.266 INFO:teuthology.orchestra.run.smithi111.stdout:4345 read 0x29d09da thru 0x29d2337 (0x195e bytes) 2024-04-23T21:22:14.266 INFO:teuthology.orchestra.run.smithi111.stdout:4346 read 0x100f649 thru 0x1010cfd (0x16b5 bytes) 2024-04-23T21:22:14.276 INFO:teuthology.orchestra.run.smithi111.stdout:4349 trunc from 0x35f172b to 0xb2abf40 2024-04-23T21:22:14.283 INFO:teuthology.orchestra.run.smithi111.stdout:4350 write 0x89b8128 thru 0x89bf410 (0x72e9 bytes) 2024-04-23T21:22:14.283 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:14.284+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:14.285 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.286 INFO:teuthology.orchestra.run.smithi111.stdout:4352 punch from 0x2eb5040 to 0x2ebefc4, (0x9f84 bytes) 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866413751000003, 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.297 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:14.301 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866390940000002, 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:14.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:14.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:14.305 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.86637133, 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:14.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:14.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:14.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866352466999999, 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:14.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:14.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866333171999997, 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.318 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866306610999999, 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:14.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.322 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866265380999998, 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:14.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:14.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.326 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866231654000003, 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:14.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866208245000003, 2024-04-23T21:22:14.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866174995000002, 2024-04-23T21:22:14.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.339 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866151602000002, 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:14.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:14.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.344 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866131052999997, 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:14.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:14.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.349 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.866105292999997, 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.350 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:14.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.865267277000001, 2024-04-23T21:22:14.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:14.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:14.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.865243946, 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.359 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.366 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.864356653000002, 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:14.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.864123073000002, 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.371 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:14.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.864006375000002, 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:14.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:14.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:14.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.863631961000003, 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.380 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:14.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.383 INFO:teuthology.orchestra.run.smithi111.stdout:4353 trunc from 0xb2abf40 to 0x239aafb 2024-04-23T21:22:14.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:14.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:14.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 44.862275076000003, 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:14.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.390 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:4357 trunc from 0x239aafb to 0x44c7449 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:4358 trunc from 0x44c7449 to 0xb991bf7 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:4359 trunc from 0xb991bf7 to 0x2de1fb 2024-04-23T21:22:14.392 INFO:teuthology.orchestra.run.smithi111.stdout:4360 write 0x2bd7d2d thru 0x2be1e9d (0xa171 bytes) 2024-04-23T21:22:14.393 INFO:teuthology.orchestra.run.smithi111.stdout:4361 write 0xcd1d72c thru 0xcd2bb70 (0xe445 bytes) 2024-04-23T21:22:14.393 INFO:teuthology.orchestra.run.smithi111.stdout:4363 writesame 0x9fd0a21 thru 0x9fdf7fb data_size 0x1(0xeddb bytes) 2024-04-23T21:22:14.395 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:14.394+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:14.418 INFO:teuthology.orchestra.run.smithi111.stdout:4364 write 0x8f23936 thru 0x8f269ee (0x30b9 bytes) 2024-04-23T21:22:14.422 INFO:teuthology.orchestra.run.smithi111.stdout:4366 punch from 0x213742d to 0x213e98f, (0x7562 bytes) 2024-04-23T21:22:14.422 INFO:teuthology.orchestra.run.smithi111.stdout:4368 write 0x9ad2543 thru 0x9add32c (0xadea bytes) 2024-04-23T21:22:14.423 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.424 INFO:teuthology.orchestra.run.smithi111.stdout:4372 read 0x8178819 thru 0x818369f (0xae87 bytes) 2024-04-23T21:22:14.424 INFO:teuthology.orchestra.run.smithi111.stdout:4375 read 0xa31de6f thru 0xa325145 (0x72d7 bytes) 2024-04-23T21:22:14.424 INFO:teuthology.orchestra.run.smithi111.stdout:4377 punch from 0xb903d5c to 0xb9042ff, (0x5a3 bytes) 2024-04-23T21:22:14.432 INFO:teuthology.orchestra.run.smithi111.stdout:4378 trunc from 0xcd2bb71 to 0x61fb361 2024-04-23T21:22:14.444 INFO:teuthology.orchestra.run.smithi111.stdout:4379 write 0x605831 thru 0x608dd2 (0x35a2 bytes) 2024-04-23T21:22:14.451 INFO:teuthology.orchestra.run.smithi111.stdout:4380 trunc from 0x61fb361 to 0xa3aa4db 2024-04-23T21:22:14.458 INFO:teuthology.orchestra.run.smithi111.stdout:4381 writesame 0x47ce28f thru 0x47da0b9 data_size 0x1(0xbe2b bytes) 2024-04-23T21:22:14.490 INFO:teuthology.orchestra.run.smithi111.stdout:4382 trunc from 0xa3aa4db to 0xf4d9d2 2024-04-23T21:22:14.495 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:14.495+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:14.515 INFO:teuthology.orchestra.run.smithi111.stdout:4385 writesame 0x67978d2 thru 0x679d07f data_size 0x57(0x57ae bytes) 2024-04-23T21:22:14.515 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.515 INFO:teuthology.orchestra.run.smithi111.stdout:4388 read 0x109ba11 thru 0x10a2f8c (0x757c bytes) 2024-04-23T21:22:14.515 INFO:teuthology.orchestra.run.smithi111.stdout:4390 flatten 2024-04-23T21:22:14.516 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.517 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:14.517 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:14.517 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.522 INFO:teuthology.orchestra.run.smithi111.stdout:4391 writesame 0x4bcdbf3 thru 0x4bda2b1 data_size 0x1(0xc6bf bytes) 2024-04-23T21:22:14.526 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:14.552 INFO:teuthology.orchestra.run.smithi111.stdout:4392 write 0x8806b8d thru 0x880b764 (0x4bd8 bytes) 2024-04-23T21:22:14.555 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.555 INFO:teuthology.orchestra.run.smithi111.stdout:4394 read 0x6d2a6d4 thru 0x6d308cf (0x61fc bytes) 2024-04-23T21:22:14.556 INFO:teuthology.orchestra.run.smithi111.stdout:4395 write 0x3d6eb69 thru 0x3d7d7a6 (0xec3e bytes) 2024-04-23T21:22:14.573 INFO:teuthology.orchestra.run.smithi111.stdout:4396 write 0xe16e6df thru 0xe17056a (0x1e8c bytes) 2024-04-23T21:22:14.576 INFO:teuthology.orchestra.run.smithi111.stdout:4397 read 0x1e9ed12 thru 0x1ea5d37 (0x7026 bytes) 2024-04-23T21:22:14.576 INFO:teuthology.orchestra.run.smithi111.stdout:4398 writesame 0x4c71251 thru 0x4c7566c data_size 0x6(0x441c bytes) 2024-04-23T21:22:14.584 INFO:teuthology.orchestra.run.smithi111.stdout:4400 write 0xe8a1a3b thru 0xe8a8c29 (0x71ef bytes) 2024-04-23T21:22:14.587 INFO:teuthology.orchestra.run.smithi111.stdout:4401 write 0x6219d1 thru 0x628de7 (0x7417 bytes) 2024-04-23T21:22:14.589 INFO:teuthology.orchestra.run.smithi111.stdout:4402 write 0x3dd72ef thru 0x3ddc59e (0x52b0 bytes) 2024-04-23T21:22:14.590 INFO:teuthology.orchestra.run.smithi111.stdout:4403 punch from 0xd24219c to 0xd2499bd, (0x7821 bytes) 2024-04-23T21:22:14.590 INFO:teuthology.orchestra.run.smithi111.stdout:4404 write 0x42ecadb thru 0x42f1be9 (0x510f bytes) 2024-04-23T21:22:14.592 INFO:teuthology.orchestra.run.smithi111.stdout:4405 read 0xa2d56e9 thru 0xa2daa19 (0x5331 bytes) 2024-04-23T21:22:14.592 INFO:teuthology.orchestra.run.smithi111.stdout:4406 read 0x38b095a thru 0x38b9094 (0x873b bytes) 2024-04-23T21:22:14.592 INFO:teuthology.orchestra.run.smithi111.stdout:4407 read 0xedd1cd thru 0xee84e1 (0xb315 bytes) 2024-04-23T21:22:14.596 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:14.596+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:14.602 INFO:teuthology.orchestra.run.smithi111.stdout:4408 trunc from 0xe8a8c2a to 0x60e269e 2024-04-23T21:22:14.614 INFO:teuthology.orchestra.run.smithi111.stdout:4409 read 0x50945f7 thru 0x50a3ded (0xf7f7 bytes) 2024-04-23T21:22:14.623 INFO:teuthology.orchestra.run.smithi111.stdout:4410 writesame 0x818e0cc thru 0x819524c data_size 0x7(0x7181 bytes) 2024-04-23T21:22:14.630 INFO:teuthology.orchestra.run.smithi111.stdout:4411 write 0x913742c thru 0x9142fc7 (0xbb9c bytes) 2024-04-23T21:22:14.633 INFO:teuthology.orchestra.run.smithi111.stdout:4412 punch from 0x5343432 to 0x534566a, (0x2238 bytes) 2024-04-23T21:22:14.634 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:14.634 INFO:teuthology.orchestra.run.smithi111.stdout:4414 read 0x8b53d48 thru 0x8b5f8d4 (0xbb8d bytes) 2024-04-23T21:22:14.634 INFO:teuthology.orchestra.run.smithi111.stdout:4415 write 0x14db812 thru 0x14dc378 (0xb67 bytes) 2024-04-23T21:22:14.635 INFO:teuthology.orchestra.run.smithi111.stdout:4418 write 0x71a08a thru 0x7286a4 (0xe61b bytes) 2024-04-23T21:22:14.638 INFO:teuthology.orchestra.run.smithi111.stdout:4419 punch from 0x357304d to 0x357e334, (0xb2e7 bytes) 2024-04-23T21:22:14.638 INFO:teuthology.orchestra.run.smithi111.stdout:4420 clone 37 order 23 su 524288 sc 12 2024-04-23T21:22:14.663 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.663 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:14.663 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:14.663 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:14.663 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:14.714 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:14.714+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:14.798 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:14.799+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.586078964999999, 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.817 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:14.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.586049297999999, 2024-04-23T21:22:14.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:14.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:14.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:14.825 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.586029889999999, 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.826 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.586008081999999, 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.830 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.585988479000001, 2024-04-23T21:22:14.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.838 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.585968873999999, 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:14.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.585948033000001, 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.846 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.585918767999999, 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:14.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:14.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.585887575000001, 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:14.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.777804564, 2024-04-23T21:22:14.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:14.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774576346, 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:14.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:14.864 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774553964999999, 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:14.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:14.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774534686999999, 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.869 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774517252999999, 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.873 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:14.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:14.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774499803000001, 2024-04-23T21:22:14.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.881 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774482427999999, 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:14.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:14.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774464385999998, 2024-04-23T21:22:14.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:14.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:14.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.890 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.774439265000002, 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.891 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:14.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 26.772319693, 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:14.895 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:14.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 14.432696701999999, 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:14.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:14.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:14.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:14.904 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:14.906 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:14.906 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:14.906+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:15.006 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:15.007+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:15.048 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:15.049 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:15.049 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:15.049 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:15.057 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:15.141 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:15.141+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:15.200 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:15.200 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:15.200 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:15.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:15.209 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:15.209+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:15.310 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:15.310+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:15.360 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:15.360 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.720535135999999, 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:15.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:15.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:15.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:15.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.720503029, 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.365 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.366 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:15.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.720479643000001, 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:15.369 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.720459336000001, 2024-04-23T21:22:15.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:15.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:15.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.720439524, 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.378 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:15.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.381 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719368125999999, 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.382 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:15.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:15.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:15.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719334781000001, 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.386 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.387 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719303252, 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.390 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:15.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719275863, 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.394 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:15.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:15.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719263956999999, 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:15.398 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.402 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719245712999999, 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:15.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719235896000001, 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.407 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.410 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719225648999998, 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:15.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:15.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719215997999999, 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.719182498999999, 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.419 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:15.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:15.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:15.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.715557075, 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.423 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:15.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.715539205999999, 2024-04-23T21:22:15.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.715520476999998, 2024-04-23T21:22:15.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:15.435 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.715502049000001, 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:15.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.439 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 28.715484144000001, 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:15.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:15.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:15.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:15.443 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:15.445 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:15.445 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-04-23T21:22:15.446 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:15.445+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:15.508 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.508 INFO:teuthology.orchestra.run.smithi111.stdout:4423 punch from 0x8f714c to 0x8f7c39, (0xaed bytes) 2024-04-23T21:22:15.512 INFO:teuthology.orchestra.run.smithi111.stdout:4425 writesame 0x26aaf9c thru 0x26b168e data_size 0x23(0x66f3 bytes) 2024-04-23T21:22:15.521 INFO:teuthology.orchestra.run.smithi111.stdout:4426 read 0x44c82be thru 0x44cc57f (0x42c2 bytes) 2024-04-23T21:22:15.521 INFO:teuthology.orchestra.run.smithi111.stdout:4427 read 0xeebb29 thru 0xef1f87 (0x645f bytes) 2024-04-23T21:22:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:4429 read 0x22de05b thru 0x22eae9a (0xce40 bytes) 2024-04-23T21:22:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:4430 punch from 0x831f376 to 0x8325162, (0x5dec bytes) 2024-04-23T21:22:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:4431 read 0x60dd529 thru 0x60e9f87 (0xca5f bytes) 2024-04-23T21:22:15.522 INFO:teuthology.orchestra.run.smithi111.stdout:4432 read 0x35cc1b6 thru 0x35d3063 (0x6eae bytes) 2024-04-23T21:22:15.523 INFO:teuthology.orchestra.run.smithi111.stdout:4434 read 0x1b9db24 thru 0x1b9f214 (0x16f1 bytes) 2024-04-23T21:22:15.523 INFO:teuthology.orchestra.run.smithi111.stdout:4435 read 0x2644eb8 thru 0x26533ff (0xe548 bytes) 2024-04-23T21:22:15.524 INFO:teuthology.orchestra.run.smithi111.stdout:4436 punch from 0x5a5d3f6 to 0x5a62535, (0x513f bytes) 2024-04-23T21:22:15.524 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.524 INFO:teuthology.orchestra.run.smithi111.stdout:4439 trunc from 0x9142fc8 to 0x94344c8 2024-04-23T21:22:15.535 INFO:teuthology.orchestra.run.smithi111.stdout:4440 trunc from 0x94344c8 to 0xd65879d 2024-04-23T21:22:15.541 INFO:teuthology.orchestra.run.smithi111.stdout:4441 read 0x78faf03 thru 0x7908aa4 (0xdba2 bytes) 2024-04-23T21:22:15.542 INFO:teuthology.orchestra.run.smithi111.stdout:4444 punch from 0xc7c9453 to 0xc7d8ecc, (0xfa79 bytes) 2024-04-23T21:22:15.542 INFO:teuthology.orchestra.run.smithi111.stdout:4446 writesame 0x88be4f3 thru 0x88c40b3 data_size 0x1(0x5bc1 bytes) 2024-04-23T21:22:15.545 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:15.546+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:15.557 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.557 INFO:teuthology.orchestra.run.smithi111.stdout:4449 read 0xbc419c9 thru 0xbc4a275 (0x88ad bytes) 2024-04-23T21:22:15.558 INFO:teuthology.orchestra.run.smithi111.stdout:4451 read 0x2c74dff thru 0x2c80b13 (0xbd15 bytes) 2024-04-23T21:22:15.558 INFO:teuthology.orchestra.run.smithi111.stdout:4452 write 0x1248f90 thru 0x1252e9c (0x9f0d bytes) 2024-04-23T21:22:15.567 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:15.567 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:15.567 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:15.567 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:15.568 INFO:teuthology.orchestra.run.smithi111.stdout:4454 write 0xa1303b7 thru 0xa13a7bd (0xa407 bytes) 2024-04-23T21:22:15.573 INFO:teuthology.orchestra.run.smithi111.stdout:4455 read 0xb0f74fc thru 0xb0f87e0 (0x12e5 bytes) 2024-04-23T21:22:15.573 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.573 INFO:teuthology.orchestra.run.smithi111.stdout:4457 read 0xaa3f09d thru 0xaa419ae (0x2912 bytes) 2024-04-23T21:22:15.574 INFO:teuthology.orchestra.run.smithi111.stdout:4458 write 0x6854b39 thru 0x685657e (0x1a46 bytes) 2024-04-23T21:22:15.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:15.578 INFO:teuthology.orchestra.run.smithi111.stdout:4459 write 0x5943e12 thru 0x5944b3c (0xd2b bytes) 2024-04-23T21:22:15.584 INFO:teuthology.orchestra.run.smithi111.stdout:4465 write 0xb01ac11 thru 0xb028a98 (0xde88 bytes) 2024-04-23T21:22:15.586 INFO:teuthology.orchestra.run.smithi111.stdout:4466 punch from 0x39c9620 to 0x39ce555, (0x4f35 bytes) 2024-04-23T21:22:15.586 INFO:teuthology.orchestra.run.smithi111.stdout:4468 writesame 0xd6138d8 thru 0xd61b45f data_size 0x43(0x7b88 bytes) 2024-04-23T21:22:15.589 INFO:teuthology.orchestra.run.smithi111.stdout:4470 writesame 0x34dc344 thru 0x34e5fa2 data_size 0x1(0x9c5f bytes) 2024-04-23T21:22:15.614 INFO:teuthology.orchestra.run.smithi111.stdout:4471 read 0x70c957 thru 0x713465 (0x6b0f bytes) 2024-04-23T21:22:15.617 INFO:teuthology.orchestra.run.smithi111.stdout:4472 trunc from 0xd65879d to 0xba7cdcb 2024-04-23T21:22:15.645 INFO:teuthology.orchestra.run.smithi111.stdout:4473 read 0x8cc2239 thru 0x8ccff40 (0xdd08 bytes) 2024-04-23T21:22:15.645 INFO:teuthology.orchestra.run.smithi111.stdout:4476 writesame 0x8badfa6 thru 0x8bb6df3 data_size 0x5(0x8e4e bytes) 2024-04-23T21:22:15.650 INFO:teuthology.orchestra.run.smithi111.stdout:4477 trunc from 0xba7cdcb to 0xb0bb284 2024-04-23T21:22:15.661 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:15.662+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:15.663 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.663 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:15.668 INFO:teuthology.orchestra.run.smithi111.stdout:4480 trunc from 0xb0bb284 to 0x64edfa1 2024-04-23T21:22:15.694 INFO:teuthology.orchestra.run.smithi111.stdout:4482 write 0x95bc6d1 thru 0x95c9a14 (0xd344 bytes) 2024-04-23T21:22:15.697 INFO:teuthology.orchestra.run.smithi111.stdout:4483 write 0x18b9791 thru 0x18c12c4 (0x7b34 bytes) 2024-04-23T21:22:15.707 INFO:teuthology.orchestra.run.smithi111.stdout:4484 read 0x920a45 thru 0x9272c8 (0x6884 bytes) 2024-04-23T21:22:15.708 INFO:teuthology.orchestra.run.smithi111.stdout:4485 punch from 0x7f054dc to 0x7f105c9, (0xb0ed bytes) 2024-04-23T21:22:15.708 INFO:teuthology.orchestra.run.smithi111.stdout:4486 clone 38 order 23 su 524288 sc 13 2024-04-23T21:22:15.733 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:15.733 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:15.734 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:15.734 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:15.734 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:15.742 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:15.783 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:15.784+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:15.847 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:15.848+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:15.886 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:15.886 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:15.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:15.886 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.930009458000001, 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:15.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:15.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:15.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929971338000001, 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:15.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:15.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:15.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929948938999999, 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.896 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:15.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:15.900 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929929543, 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:15.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.904 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929909944999999, 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:15.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929889731999999, 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:15.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929857565999999, 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.913 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:15.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:15.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929832953000002, 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:15.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:15.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929811955000002, 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:15.922 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:15.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929791655999999, 2024-04-23T21:22:15.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:15.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:15.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.929713867, 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:15.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:15.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.928218492999999, 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.928168603, 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.940 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:15.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.928144834000001, 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:15.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:15.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.928123828, 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:15.948 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.952 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.928104222999998, 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:15.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:15.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:15.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.912102074, 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.957 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.912083562999999, 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:15.961 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:15.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:15.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.912063117999999, 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:15.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:15.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.932985905999999, 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:15.970 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:15.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.976 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:15.976 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:15.976 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:15.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:15.978 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:15.978 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:15.978+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:16.078 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:16.078+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:16.095 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:16.096 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:16.112 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:16.113 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:16.122 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:16.122 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:16.122 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:16.122 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:16.129 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:16.129 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:16.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:16.146 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:16.146 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:16.163 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:16.163 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:16.179 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:16.180+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:16.180 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:16.180 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:16.197 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:16.197 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:16.214 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:16.214 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:16.234 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:16.234 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:16.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:16.258 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:22:16.286 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:16.287+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:16.308 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone37 from 0x95c9a15 (overlap 0x64edfa1) to 0x1e47649 2024-04-23T21:22:16.374 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:16.374 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:16.380 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:16.381+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:16.384 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:16.429 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-04-23T21:22:16.481 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:16.482+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:16.529 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:16.529 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:16.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:16.591 INFO:teuthology.orchestra.run.smithi111.stdout:4487 read 0x11b5342 thru 0x11c01ca (0xae89 bytes) 2024-04-23T21:22:16.593 INFO:teuthology.orchestra.run.smithi111.stdout:4488 trunc from 0x95c9a15 to 0x8856d34 2024-04-23T21:22:16.619 INFO:teuthology.orchestra.run.smithi111.stdout:4489 punch from 0x6e5f6c6 to 0x6e60226, (0xb60 bytes) 2024-04-23T21:22:16.620 INFO:teuthology.orchestra.run.smithi111.stdout:4490 trunc from 0x8856d34 to 0x7652f4d 2024-04-23T21:22:16.622 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:16.622+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:16.628 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:16.633 INFO:teuthology.orchestra.run.smithi111.stdout:4492 write 0x7d04837 thru 0x7d142ce (0xfa98 bytes) 2024-04-23T21:22:16.635 INFO:teuthology.orchestra.run.smithi111.stdout:4493 punch from 0x52575a4 to 0x5259e7d, (0x28d9 bytes) 2024-04-23T21:22:16.635 INFO:teuthology.orchestra.run.smithi111.stdout:4494 read 0x55d2f28 thru 0x55dd8e5 (0xa9be bytes) 2024-04-23T21:22:16.635 INFO:teuthology.orchestra.run.smithi111.stdout:4495 write 0x62308a2 thru 0x62375c5 (0x6d24 bytes) 2024-04-23T21:22:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:4496 punch from 0x172c043 to 0x1734a91, (0x8a4e bytes) 2024-04-23T21:22:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:4497 read 0x614d502 thru 0x6156dff (0x98fe bytes) 2024-04-23T21:22:16.645 INFO:teuthology.orchestra.run.smithi111.stdout:4498 read 0xc6a49d thru 0xc6b5a1 (0x1105 bytes) 2024-04-23T21:22:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:4499 read 0x236b514 thru 0x2377744 (0xc231 bytes) 2024-04-23T21:22:16.646 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:16.654 INFO:teuthology.orchestra.run.smithi111.stdout:4502 write 0xa2cfb6a thru 0xa2d4834 (0x4ccb bytes) 2024-04-23T21:22:16.662 INFO:teuthology.orchestra.run.smithi111.stdout:4503 write 0xae6a6a1 thru 0xae75237 (0xab97 bytes) 2024-04-23T21:22:16.664 INFO:teuthology.orchestra.run.smithi111.stdout:4506 punch from 0xaa936f8 to 0xaa99857, (0x615f bytes) 2024-04-23T21:22:16.664 INFO:teuthology.orchestra.run.smithi111.stdout:4507 read 0x59d818c thru 0x59dd4de (0x5353 bytes) 2024-04-23T21:22:16.664 INFO:teuthology.orchestra.run.smithi111.stdout:4510 write 0x6e922cf thru 0x6ea1876 (0xf5a8 bytes) 2024-04-23T21:22:16.666 INFO:teuthology.orchestra.run.smithi111.stdout:4512 writesame 0x9418180 thru 0x9425ae6 data_size 0x1(0xd967 bytes) 2024-04-23T21:22:16.682 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:16.683+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:16.700 INFO:teuthology.orchestra.run.smithi111.stdout:4513 write 0x456b3fe thru 0x4571b05 (0x6708 bytes) 2024-04-23T21:22:16.702 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:16.703 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:16.714 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:16.714 INFO:teuthology.orchestra.run.smithi111.stdout:4514 writesame 0xb80b229 thru 0xb81533c data_size 0x3d(0xa114 bytes) 2024-04-23T21:22:16.722 INFO:teuthology.orchestra.run.smithi111.stdout:4516 write 0xd40591a thru 0xd407523 (0x1c0a bytes) 2024-04-23T21:22:16.725 INFO:teuthology.orchestra.run.smithi111.stdout:4517 read 0x99e10ff thru 0x99ee9dc (0xd8de bytes) 2024-04-23T21:22:16.725 INFO:teuthology.orchestra.run.smithi111.stdout:4518 writesame 0x5467c1b thru 0x546ef9f data_size 0x1(0x7385 bytes) 2024-04-23T21:22:16.744 INFO:teuthology.orchestra.run.smithi111.stdout:4519 punch from 0xc4a01e0 to 0xc4a132f, (0x114f bytes) 2024-04-23T21:22:16.744 INFO:teuthology.orchestra.run.smithi111.stdout:4520 punch from 0x3a1b902 to 0x3a292d9, (0xd9d7 bytes) 2024-04-23T21:22:16.744 INFO:teuthology.orchestra.run.smithi111.stdout:4521 writesame 0x9b53aac thru 0x9b61b0b data_size 0x28(0xe060 bytes) 2024-04-23T21:22:16.745 INFO:teuthology.orchestra.run.smithi111.stdout:4522 read 0x6675558 thru 0x667e209 (0x8cb2 bytes) 2024-04-23T21:22:16.746 INFO:teuthology.orchestra.run.smithi111.stdout:4523 read 0xb618788 thru 0xb61c56f (0x3de8 bytes) 2024-04-23T21:22:16.746 INFO:teuthology.orchestra.run.smithi111.stdout:4524 read 0x9e6c6c4 thru 0x9e6eff4 (0x2931 bytes) 2024-04-23T21:22:16.746 INFO:teuthology.orchestra.run.smithi111.stdout:4525 write 0x4b49f1e thru 0x4b4db0e (0x3bf1 bytes) 2024-04-23T21:22:16.750 INFO:teuthology.orchestra.run.smithi111.stdout:4526 write 0x62cf83c thru 0x62d1fef (0x27b4 bytes) 2024-04-23T21:22:16.759 INFO:teuthology.orchestra.run.smithi111.stdout:4527 read 0xb715cfe thru 0xb724368 (0xe66b bytes) 2024-04-23T21:22:16.760 INFO:teuthology.orchestra.run.smithi111.stdout:4528 read 0x1dd869d thru 0x1de80d7 (0xfa3b bytes) 2024-04-23T21:22:16.760 INFO:teuthology.orchestra.run.smithi111.stdout:4530 writesame 0xcb5860b thru 0xcb5bd06 data_size 0x33(0x36fc bytes) 2024-04-23T21:22:16.761 INFO:teuthology.orchestra.run.smithi111.stdout:4533 write 0xc794942 thru 0xc79af80 (0x663f bytes) 2024-04-23T21:22:16.763 INFO:teuthology.orchestra.run.smithi111.stdout:4534 writesame 0x7425236 thru 0x7428db3 data_size 0x5(0x3b7e bytes) 2024-04-23T21:22:16.765 INFO:teuthology.orchestra.run.smithi111.stdout:4535 read 0x51a002d thru 0x51a56a5 (0x5679 bytes) 2024-04-23T21:22:16.765 INFO:teuthology.orchestra.run.smithi111.stdout:4536 punch from 0x4cde662 to 0x4ce3353, (0x4cf1 bytes) 2024-04-23T21:22:16.765 INFO:teuthology.orchestra.run.smithi111.stdout:4537 clone 39 order 22 su 131072 sc 4 2024-04-23T21:22:16.800 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:16.801+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:16.859 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:16.859 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:16.867 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:16.884 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:16.884+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:16.985 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:16.985+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:17.009 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.010 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:17.086 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:17.086+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:17.120 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.120 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:17.187 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:17.187+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:17.232 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.232 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.241 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:17.257 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:22:17.287 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:17.288+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:17.317 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone38 from 0xd407524 (overlap 0x7652f4d) to 0x5ea8c3d 2024-04-23T21:22:17.364 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.365 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.374 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:17.388 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:17.389+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:17.474 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-04-23T21:22:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.484 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.490 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:17.490+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:17.496 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:17.514 INFO:teuthology.orchestra.run.smithi111.stdout:4538 writesame 0xa9aad79 thru 0xa9b78c6 data_size 0x35(0xcb4e bytes) 2024-04-23T21:22:17.523 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:17.523 INFO:teuthology.orchestra.run.smithi111.stdout:4540 read 0x42f8295 thru 0x42fc81e (0x458a bytes) 2024-04-23T21:22:17.523 INFO:teuthology.orchestra.run.smithi111.stdout:4541 read 0x23799bd thru 0x2389636 (0xfc7a bytes) 2024-04-23T21:22:17.524 INFO:teuthology.orchestra.run.smithi111.stdout:4542 punch from 0x6fa0814 to 0x6fa5bd7, (0x53c3 bytes) 2024-04-23T21:22:17.524 INFO:teuthology.orchestra.run.smithi111.stdout:4543 punch from 0x71e0482 to 0x71ebb3a, (0xb6b8 bytes) 2024-04-23T21:22:17.524 INFO:teuthology.orchestra.run.smithi111.stdout:4544 read 0x55ff72d thru 0x560aac7 (0xb39b bytes) 2024-04-23T21:22:17.525 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:17.525 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:17.525 INFO:teuthology.orchestra.run.smithi111.stdout:4548 clone 40 order 20 su 4096 sc 14 2024-04-23T21:22:17.591 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:17.591+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:17.617 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.617 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.626 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:17.692 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:17.693+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:17.768 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.768 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:17.822 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:17.822+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:17.879 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:17.880 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:17.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:17.929 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:17.929+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:17.995 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:17.995+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:18.031 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:18.031 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:18.040 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:18.125 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:18.125+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:18.182 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:18.182 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:18.191 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:18.197 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:18.198+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:18.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:18.297 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:18.298+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:18.335 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone39 from 0xd407524 (overlap 0xd407524) to 0x54cfba8 2024-04-23T21:22:18.349 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:18.349 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:18.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:18.381 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:18.382 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:18.409 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:18.409 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:18.409 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:18.409+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:18.427 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:18.427 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:18.446 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:18.464 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:18.465 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:18.490 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:18.490 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:18.497 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:18.497 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:18.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266454381999999, 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:18.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:18.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:18.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:18.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266424714999999, 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.502 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:18.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:18.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266405306999999, 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266383499, 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:18.510 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:18.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266363896000001, 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266344291000003, 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.519 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:18.523 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266323450000002, 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:18.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:18.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266294185, 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:18.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:18.532 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.266262992000001, 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.536 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.458179981000001, 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:18.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:18.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454951763, 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454929382, 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:18.545 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454910104, 2024-04-23T21:22:18.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:18.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:18.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.553 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.45489267, 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.557 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454875220000002, 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:18.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:18.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:18.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:18.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454857844999999, 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.562 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454839802999999, 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.566 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:18.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:18.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.454814681999999, 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:18.570 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:18.574 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 30.452695110000001, 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:18.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:18.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:18.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.113072119000002, 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.583 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:18.583 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:18.583 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:18.584 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:18.585 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:18.585 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:18.585 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:18.585 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:18.586 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-04-23T21:22:18.587 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:18.584+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:18.652 INFO:teuthology.orchestra.run.smithi111.stdout:4549 read 0x12b3049 thru 0x12bb9ca (0x8982 bytes) 2024-04-23T21:22:18.654 INFO:teuthology.orchestra.run.smithi111.stdout:4550 punch from 0xbc4073 to 0xbc8758, (0x46e5 bytes) 2024-04-23T21:22:18.658 INFO:teuthology.orchestra.run.smithi111.stdout:4551 read 0x2ce0b3d thru 0x2cea18e (0x9652 bytes) 2024-04-23T21:22:18.667 INFO:teuthology.orchestra.run.smithi111.stdout:4552 write 0xebdcbc9 thru 0xebe025e (0x3696 bytes) 2024-04-23T21:22:18.673 INFO:teuthology.orchestra.run.smithi111.stdout:4553 writesame 0x12d74e thru 0x13ca8e data_size 0x1(0xf341 bytes) 2024-04-23T21:22:18.685 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:18.685+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:18.767 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:18.767 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:18.767 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:18.767 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:18.777 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:18.785 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:18.786+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:18.846 INFO:teuthology.orchestra.run.smithi111.stdout:4555 trunc from 0xebe025f to 0x7273c61 2024-04-23T21:22:18.886 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:18.887+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:18.962 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:18.962 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:18.962 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:18.962 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:18.962 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:18.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:19.012 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:19.012+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:19.038 INFO:teuthology.orchestra.run.smithi111.stdout:4557 write 0x94d9dfd thru 0x94dae26 (0x102a bytes) 2024-04-23T21:22:19.046 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:19.046 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:19.046 INFO:teuthology.orchestra.run.smithi111.stdout:4560 read 0x4264 thru 0x4f08 (0xca5 bytes) 2024-04-23T21:22:19.053 INFO:teuthology.orchestra.run.smithi111.stdout:4561 trunc from 0x94dae27 to 0xdc083ed 2024-04-23T21:22:19.059 INFO:teuthology.orchestra.run.smithi111.stdout:4562 read 0xd653665 thru 0xd654d10 (0x16ac bytes) 2024-04-23T21:22:19.059 INFO:teuthology.orchestra.run.smithi111.stdout:4563 clone 41 order 20 su 32768 sc 15 2024-04-23T21:22:19.087 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:19.087+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.484472060000002, 2024-04-23T21:22:19.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:19.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.484439952999999, 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:19.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:19.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.132 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.484416566999997, 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:19.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:19.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:19.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.484396259999997, 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:19.138 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:19.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:19.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.143 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.484376447999999, 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:19.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:19.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483305049999998, 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.148 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:19.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.151 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:19.152 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483271705, 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:19.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:19.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483240176000002, 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:19.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483212786999999, 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:19.161 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:19.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.165 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483200881000002, 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:19.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:19.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483182636999999, 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.170 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:19.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.174 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.48317282, 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:19.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:19.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483162573000001, 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.179 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:19.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:19.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483152922000002, 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.187 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.483119422999998, 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:19.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.479493998999999, 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.192 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:19.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:19.196 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.479476130000002, 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:19.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.479457400999998, 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.479438973000001, 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.205 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:19.209 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 32.479421068000001, 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:19.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.213 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.214 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.214 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:19.214 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:19.214 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:19.214 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:19.214 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-23T21:22:19.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:19.216 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:19.216+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:19.274 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:22:19.325 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:19.325 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:19.325 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:19.325 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:19.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:19.343 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:19.343+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:19.349 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone40 from 0xdc083ed (overlap 0x7273c61) to 0x2255ee9 2024-04-23T21:22:19.418 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:19.418+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:19.487 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:19.488 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:19.488 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:19.488 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:19.488 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:19.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:19.545 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:19.546+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:19.619 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:19.619+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.760094182000003, 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.719 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:19.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.760056061999997, 2024-04-23T21:22:19.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.727 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.760033663000002, 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:19.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.731 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.760014267000003, 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.732 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:19.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:19.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:19.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759994669000001, 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759974456000002, 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.740 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759942289999998, 2024-04-23T21:22:19.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:19.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:19.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759917676999997, 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759896679000001, 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:19.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:19.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:19.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.756 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759876380000001, 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.757 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:19.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:19.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:19.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.759798590999999, 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.761 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.758303216999998, 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:19.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.758253326999998, 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.769 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:19.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.758229557999996, 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:19.773 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:19.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.777 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.758208551999999, 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.758188947000001, 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.782 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:19.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:19.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:19.785 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.742186797999999, 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.786 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.742168286999998, 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.790 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:19.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:19.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:19.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.742147842000001, 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.794 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:19.798 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.763070630000001, 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:19.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:19.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:19.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:19.803 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.804 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:19.804 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:19.804 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:19.805 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:19.806 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:19.806+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:19.906 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:19.906+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:20.007 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:20.007+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:20.027 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98364 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974452+0000", 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047713934, 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.047735687999999998, 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98364 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974452+0000", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974452+0000", 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974452+0000", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4200000000000002e-07 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974454+0000", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974459+0000", 2024-04-23T21:22:20.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7670000000000003e-06 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974463+0000", 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0060000000000003e-06 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98416 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.980980+0000", 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041185389000000003, 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041272349999999999, 2024-04-23T21:22:20.031 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98416 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980980+0000", 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980980+0000", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980981+0000", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980984+0000", 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8430000000000001e-06 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980998+0000", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4054e-05 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981003+0000", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8640000000000001e-06 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98485 3.5 3.5fb365 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.992094+0000", 2024-04-23T21:22:20.034 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.030071919999999999, 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030235608000000001, 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98485 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992094+0000", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992094+0000", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992094+0000", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.37e-07 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992098+0000", 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9450000000000003e-06 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992109+0000", 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.992117+0000", 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0749999999999998e-06 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99483 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.018058+0000", 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0041078720000000003, 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0043276369999999996, 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99483 2024-04-23T21:22:20.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018058+0000", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018058+0000", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018059+0000", 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2270000000000001e-06 2024-04-23T21:22:20.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018064+0000", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3890000000000004e-06 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018078+0000", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3890000000000001e-05 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018085+0000", 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1550000000000001e-06 2024-04-23T21:22:20.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98366 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974468+0000", 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047697874000000001, 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.047972788000000002, 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98366 2024-04-23T21:22:20.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974468+0000", 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974468+0000", 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974468+0000", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8999999999999998e-07 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974475+0000", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5719999999999999e-06 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974480+0000", 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2680000000000002e-06 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974484+0000", 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0690000000000003e-06 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98418 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981012+0000", 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041154018000000001, 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041479587999999998, 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.044 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98418 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981012+0000", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981012+0000", 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981012+0000", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9500000000000003e-07 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981014+0000", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4099999999999998e-06 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981021+0000", 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6220000000000003e-06 2024-04-23T21:22:20.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981026+0000", 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0420000000000002e-06 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98488 3.5 3.5fb365 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.993150+0000", 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.029015269999999999, 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029394413000000001, 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.047 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98488 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993150+0000", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993150+0000", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993151+0000", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6000000000000003e-07 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993208+0000", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7021999999999997e-05 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993364+0000", 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000156327 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993372+0000", 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.729e-06 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99484 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.018227+0000", 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0039384989999999998, 2024-04-23T21:22:20.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004347577, 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99484 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018227+0000", 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018227+0000", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018228+0000", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018231+0000", 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.128e-06 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018239+0000", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4719999999999995e-06 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018246+0000", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3690000000000003e-06 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98368 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974489+0000", 2024-04-23T21:22:20.053 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047676744, 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048113498999999997, 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98368 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974489+0000", 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974489+0000", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974489+0000", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2300000000000002e-07 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974491+0000", 2024-04-23T21:22:20.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0169999999999999e-06 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974496+0000", 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7709999999999997e-06 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974500+0000", 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6200000000000001e-06 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98419 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981033+0000", 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041132849999999999, 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041599338, 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98419 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981033+0000", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981033+0000", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981033+0000", 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.51e-07 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981036+0000", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981042+0000", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6080000000000001e-06 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981047+0000", 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.028e-06 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.059 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98710 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.993385+0000", 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.028780611000000001, 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029294784000000001, 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98710 2024-04-23T21:22:20.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993385+0000", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993385+0000", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993386+0000", 2024-04-23T21:22:20.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8500000000000001e-07 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993389+0000", 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7780000000000001e-06 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993401+0000", 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.152e-05 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993412+0000", 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0831e-05 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99485 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.018254+0000", 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0039114589999999999, 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044790610000000003, 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99485 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018254+0000", 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018254+0000", 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018254+0000", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018256+0000", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9149999999999999e-06 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018262+0000", 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5609999999999998e-06 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018266+0000", 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3649999999999997e-06 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98370 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974504+0000", 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047661504, 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048284834999999998, 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.066 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98370 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974504+0000", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974504+0000", 2024-04-23T21:22:20.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974504+0000", 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974506+0000", 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6169999999999999e-06 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974511+0000", 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8080000000000004e-06 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974515+0000", 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8689999999999997e-06 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98421 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981054+0000", 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041111282999999998, 2024-04-23T21:22:20.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041799891999999998, 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98421 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981054+0000", 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981054+0000", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981055+0000", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5200000000000002e-07 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981064+0000", 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0909999999999999e-06 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981074+0000", 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.045e-05 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981080+0000", 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6470000000000003e-06 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98713 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.993421+0000", 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.028744543000000001, 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029487261000000001, 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98713 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993421+0000", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993421+0000", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993422+0000", 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8400000000000005e-07 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993425+0000", 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2040000000000002e-06 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993432+0000", 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2899999999999997e-06 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.993438+0000", 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6779999999999997e-06 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98372 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974519+0000", 2024-04-23T21:22:20.076 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047646453999999998, 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048459704999999999, 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98372 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974519+0000", 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974519+0000", 2024-04-23T21:22:20.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974519+0000", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.15e-07 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974521+0000", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.517e-06 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974526+0000", 2024-04-23T21:22:20.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7709999999999997e-06 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974529+0000", 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6509999999999999e-06 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98423 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981094+0000", 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041071911000000003, 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041941025999999999, 2024-04-23T21:22:20.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98423 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981094+0000", 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981094+0000", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981094+0000", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5199999999999996e-07 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981097+0000", 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5210000000000001e-06 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981104+0000", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.25e-06 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981109+0000", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98374 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974534+0000", 2024-04-23T21:22:20.083 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047631860999999998, 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048559357999999997, 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98374 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974534+0000", 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974534+0000", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974534+0000", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974536+0000", 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.5e-06 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974540+0000", 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8160000000000001e-06 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974544+0000", 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7950000000000001e-06 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98424 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.086 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981116+0000", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041049189, 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.042032446000000001, 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98424 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981116+0000", 2024-04-23T21:22:20.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981116+0000", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981117+0000", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0999999999999999e-07 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981129+0000", 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2224e-05 2024-04-23T21:22:20.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981136+0000", 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.063e-06 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981142+0000", 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.089 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99507 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.018929+0000", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0032366270000000002, 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0042831479999999996, 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99507 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018929+0000", 2024-04-23T21:22:20.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018929+0000", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018930+0000", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799999999999995e-07 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018933+0000", 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.737e-06 2024-04-23T21:22:20.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018946+0000", 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2595999999999999e-05 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018953+0000", 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7669999999999998e-06 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.095 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98376 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.974549+0000", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.047617052999999999, 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048720686999999999, 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98376 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974549+0000", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.974549+0000", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.975931+0000", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001382753 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.975935+0000", 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9430000000000002e-06 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.975945+0000", 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0225e-05 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.975952+0000", 2024-04-23T21:22:20.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.511e-06 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98426 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981149+0000", 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.041016191, 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.042187019999999999, 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98426 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981149+0000", 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981149+0000", 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981150+0000", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8800000000000003e-07 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981152+0000", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4839999999999998e-06 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981159+0000", 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6749999999999996e-06 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981164+0000", 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1889999999999998e-06 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99508 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.018969+0000", 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0031962399999999999, 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0044236830000000003, 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.102 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99508 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018969+0000", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018969+0000", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018970+0000", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018973+0000", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9409999999999999e-06 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018981+0000", 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8679999999999999e-06 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018986+0000", 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6749999999999999e-06 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98377 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976020+0000", 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.046146063000000001, 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.047429254999999997, 2024-04-23T21:22:20.105 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98377 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976020+0000", 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976020+0000", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976020+0000", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2300000000000001e-07 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976024+0000", 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7330000000000001e-06 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976034+0000", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9189999999999994e-06 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976040+0000", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1120000000000001e-06 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98428 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.981171+0000", 2024-04-23T21:22:20.108 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.040994846000000001, 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.042334376, 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98428 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981171+0000", 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.981171+0000", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984156+0000", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0029856679999999999 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984161+0000", 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8119999999999998e-06 2024-04-23T21:22:20.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984183+0000", 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1365e-05 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984190+0000", 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6260000000000003e-06 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99536 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.111 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.022716+0000", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9994497, 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00085583999999999996, 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99536 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022716+0000", 2024-04-23T21:22:20.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022716+0000", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022717+0000", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2810000000000001e-06 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022722+0000", 2024-04-23T21:22:20.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8539999999999996e-06 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022731+0000", 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6440000000000006e-06 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022738+0000", 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8650000000000003e-06 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98378 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976049+0000", 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.046116321000000002, 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.047582336000000003, 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98378 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.115 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976049+0000", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976049+0000", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976050+0000", 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4099999999999999e-07 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976052+0000", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.323e-06 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976058+0000", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8000000000000004e-06 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976063+0000", 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7129999999999996e-06 2024-04-23T21:22:20.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98429 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984202+0000", 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037963297999999999, 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039487120000000001, 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.118 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98429 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984202+0000", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984202+0000", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984203+0000", 2024-04-23T21:22:20.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5300000000000004e-07 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984206+0000", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9189999999999999e-06 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984213+0000", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9789999999999996e-06 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984218+0000", 2024-04-23T21:22:20.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0479999999999998e-06 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98733 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994371+0000", 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027794694000000002, 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029375968999999998, 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98733 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994371+0000", 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994371+0000", 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994372+0000", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5199999999999996e-07 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994375+0000", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7680000000000002e-06 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994383+0000", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3420000000000002e-06 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994389+0000", 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.981e-06 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99539 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.022750+0000", 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9994154, 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001065463, 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99539 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022750+0000", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022750+0000", 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022751+0000", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.13e-07 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022753+0000", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3549999999999999e-06 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022760+0000", 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.022765+0000", 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2220000000000001e-06 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98379 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976070+0000", 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.04609572, 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.047809863000000001, 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.127 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98379 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976070+0000", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976070+0000", 2024-04-23T21:22:20.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976070+0000", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976072+0000", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1950000000000002e-06 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976078+0000", 2024-04-23T21:22:20.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6790000000000002e-06 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976083+0000", 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4109999999999998e-06 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98431 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984226+0000", 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037940082, 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039713904000000001, 2024-04-23T21:22:20.130 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98431 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984226+0000", 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984226+0000", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984226+0000", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.96e-07 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984228+0000", 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1940000000000001e-06 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984234+0000", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.173e-06 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984239+0000", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8389999999999997e-06 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98734 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994399+0000", 2024-04-23T21:22:20.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027766830999999999, 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029602999000000001, 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98734 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994399+0000", 2024-04-23T21:22:20.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994399+0000", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994399+0000", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.01e-07 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994401+0000", 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2390000000000001e-06 2024-04-23T21:22:20.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994407+0000", 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994412+0000", 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7690000000000004e-06 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.136 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:98247 3.5 e75/18 ondisk, result = 0)", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.023682+0000", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9984837, 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018995199999999999, 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023682+0000", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023682+0000", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023683+0000", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4200000000000005e-07 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023688+0000", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9189999999999998e-06 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023698+0000", 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0069e-05 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023704+0000", 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7309999999999996e-06 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023769+0000", 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4696000000000005e-05 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023776+0000", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7859999999999999e-06 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.023872+0000", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5909000000000006e-05 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98380 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976088+0000", 2024-04-23T21:22:20.140 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.046077313000000002, 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048050746999999998, 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98380 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976088+0000", 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976088+0000", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976089+0000", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976091+0000", 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1040000000000001e-06 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976097+0000", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2829999999999998e-06 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976102+0000", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1050000000000003e-06 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98433 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984246+0000", 2024-04-23T21:22:20.143 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037919710000000002, 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039949743000000003, 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98433 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984246+0000", 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984246+0000", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984246+0000", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0399999999999996e-07 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984248+0000", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984255+0000", 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5259999999999999e-06 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984260+0000", 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8099999999999997e-06 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98763 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994418+0000", 2024-04-23T21:22:20.146 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027747225, 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029832477, 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98763 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994418+0000", 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994418+0000", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994419+0000", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994421+0000", 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1330000000000002e-06 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994427+0000", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.596e-06 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994432+0000", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5680000000000001e-06 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98381 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976108+0000", 2024-04-23T21:22:20.149 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.046057459000000002, 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048198195999999999, 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98381 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976108+0000", 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976108+0000", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976108+0000", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976110+0000", 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.88e-06 2024-04-23T21:22:20.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976117+0000", 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.263e-06 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976121+0000", 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5379999999999996e-06 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98435 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984276+0000", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037889549000000002, 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040086399000000002, 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98435 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984276+0000", 2024-04-23T21:22:20.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984276+0000", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984277+0000", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984286+0000", 2024-04-23T21:22:20.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0319999999999992e-06 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984293+0000", 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2010000000000002e-06 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984298+0000", 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9980000000000003e-06 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98764 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994438+0000", 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027727946999999999, 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029980071, 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98764 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994438+0000", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994438+0000", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994438+0000", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.27e-07 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994440+0000", 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1090000000000001e-06 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994544+0000", 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104189 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994554+0000", 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6080000000000002e-06 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98382 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.976127+0000", 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.046038411000000001, 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.048345733000000002, 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98382 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976127+0000", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.976127+0000", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978772+0000", 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0026452450000000001 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978778+0000", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1189999999999996e-06 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978790+0000", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2109999999999999e-05 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978797+0000", 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0149999999999998e-06 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98436 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984313+0000", 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037852627, 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040215437999999999, 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98436 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.162 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984313+0000", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984313+0000", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984313+0000", 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4900000000000001e-07 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984316+0000", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6280000000000001e-06 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984323+0000", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6669999999999999e-06 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984327+0000", 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4129999999999999e-06 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98967 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994565+0000", 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.02760106, 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030018802000000001, 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98967 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.165 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994565+0000", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994565+0000", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994565+0000", 2024-04-23T21:22:20.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994574+0000", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3440000000000007e-06 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994583+0000", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3310000000000004e-06 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994588+0000", 2024-04-23T21:22:20.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3340000000000001e-06 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98383 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.978810+0000", 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043355991000000003, 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.045829011000000003, 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98383 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978810+0000", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978810+0000", 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978810+0000", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978823+0000", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2653000000000001e-05 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978831+0000", 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1769999999999998e-06 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978837+0000", 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0120000000000002e-06 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98437 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984333+0000", 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037832230000000001, 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040364338, 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98437 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984333+0000", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984333+0000", 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984334+0000", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4999999999999998e-07 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984336+0000", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984342+0000", 2024-04-23T21:22:20.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6339999999999998e-06 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984346+0000", 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.206e-06 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98968 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994595+0000", 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027570543999999999, 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030157100999999999, 2024-04-23T21:22:20.174 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98968 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994595+0000", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994595+0000", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994595+0000", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0400000000000002e-07 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994597+0000", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.951e-06 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994604+0000", 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1090000000000003e-06 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994608+0000", 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7910000000000004e-06 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98385 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.978846+0000", 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043319502000000003, 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.045973435999999999, 2024-04-23T21:22:20.177 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98385 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978846+0000", 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978846+0000", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978847+0000", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978849+0000", 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5210000000000001e-06 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978874+0000", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4620000000000001e-05 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978892+0000", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7929000000000001e-05 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98438 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984352+0000", 2024-04-23T21:22:20.180 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037814017999999998, 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040543797999999999, 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98438 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984352+0000", 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984352+0000", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984352+0000", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984354+0000", 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9709999999999998e-06 2024-04-23T21:22:20.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984360+0000", 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8930000000000002e-06 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984364+0000", 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5360000000000003e-06 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98969 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.183 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.994614+0000", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027551425000000001, 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030343931000000001, 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98969 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994614+0000", 2024-04-23T21:22:20.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994614+0000", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994615+0000", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.89e-07 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994616+0000", 2024-04-23T21:22:20.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8220000000000001e-06 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994622+0000", 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.994627+0000", 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6909999999999996e-06 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98387 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.978901+0000", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043264572000000001, 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046113370000000001, 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98387 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978901+0000", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978901+0000", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978902+0000", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6400000000000002e-07 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978904+0000", 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8899999999999999e-06 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978912+0000", 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.554e-06 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978919+0000", 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0210000000000002e-06 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98439 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984372+0000", 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037794083999999999, 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040709447000000003, 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98439 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.190 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984372+0000", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984372+0000", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984372+0000", 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0400000000000002e-07 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984374+0000", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3250000000000002e-06 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984380+0000", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.105e-06 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984390+0000", 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9559999999999992e-06 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.192 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98389 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.978934+0000", 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043232021000000002, 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046196777000000001, 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98389 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978934+0000", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978934+0000", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978934+0000", 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5200000000000002e-07 2024-04-23T21:22:20.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978936+0000", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.463e-06 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978950+0000", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3586e-05 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978956+0000", 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1279999999999996e-06 2024-04-23T21:22:20.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98440 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984407+0000", 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037758702999999998, 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040758569000000001, 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98440 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984407+0000", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984407+0000", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984407+0000", 2024-04-23T21:22:20.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4900000000000001e-07 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984410+0000", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.509e-06 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984416+0000", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6259999999999997e-06 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984421+0000", 2024-04-23T21:22:20.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8949999999999997e-06 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98391 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.978978+0000", 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043187656999999997, 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046216477999999998, 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.199 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98391 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978978+0000", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978978+0000", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978978+0000", 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.82e-07 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978981+0000", 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.593e-06 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978989+0000", 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9519999999999994e-06 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.978995+0000", 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8549999999999998e-06 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98441 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984428+0000", 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037737582999999998, 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040793746999999998, 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.202 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98441 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984428+0000", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984428+0000", 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984428+0000", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7899999999999999e-07 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984435+0000", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2999999999999998e-06 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984441+0000", 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1550000000000004e-06 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984445+0000", 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2470000000000002e-06 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98990 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.995080+0000", 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.027085332, 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030169055, 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98990 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995080+0000", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995080+0000", 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995081+0000", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9299999999999998e-07 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995084+0000", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2030000000000001e-06 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995091+0000", 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2089999999999999e-06 2024-04-23T21:22:20.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995097+0000", 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0419999999999999e-06 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98442 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984451+0000", 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037714545000000002, 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040825669000000002, 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.208 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98442 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984451+0000", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984451+0000", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984452+0000", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984453+0000", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9309999999999998e-06 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984459+0000", 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3730000000000001e-06 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984463+0000", 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4469999999999999e-06 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98992 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.995178+0000", 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.026987365999999999, 2024-04-23T21:22:20.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030125923999999998, 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98992 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995178+0000", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995178+0000", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995179+0000", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0800000000000004e-07 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995182+0000", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4130000000000002e-06 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995192+0000", 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4180000000000003e-06 2024-04-23T21:22:20.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995198+0000", 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8159999999999996e-06 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98393 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979029+0000", 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043136675999999999, 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046302492000000001, 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.214 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98393 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979029+0000", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979029+0000", 2024-04-23T21:22:20.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979029+0000", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8299999999999997e-07 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979032+0000", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5160000000000001e-06 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979041+0000", 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4930000000000004e-06 2024-04-23T21:22:20.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979047+0000", 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98443 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984469+0000", 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037696273000000002, 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040893454000000003, 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.217 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98443 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984469+0000", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984469+0000", 2024-04-23T21:22:20.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984470+0000", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984472+0000", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0789999999999999e-06 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984477+0000", 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2460000000000003e-06 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984481+0000", 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1749999999999998e-06 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99020 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.995208+0000", 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.026957095, 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030182203000000001, 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.220 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99020 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995208+0000", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995208+0000", 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995209+0000", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995211+0000", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3549999999999999e-06 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995218+0000", 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7429999999999996e-06 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995224+0000", 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98445 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984487+0000", 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037678876, 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040931707999999997, 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.223 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98445 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984487+0000", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984487+0000", 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984487+0000", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3999999999999997e-07 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984489+0000", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.869e-06 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984494+0000", 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.361e-06 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984499+0000", 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3819999999999997e-06 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99021 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.995231+0000", 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.026934637000000001, 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.030215057999999999, 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.226 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99021 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995231+0000", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995231+0000", 2024-04-23T21:22:20.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995231+0000", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995233+0000", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0870000000000001e-06 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995240+0000", 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1399999999999997e-06 2024-04-23T21:22:20.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.995246+0000", 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6800000000000004e-06 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98395 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979160+0000", 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.043005754, 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046314307999999998, 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.229 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98395 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979160+0000", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979160+0000", 2024-04-23T21:22:20.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979160+0000", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979163+0000", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2170000000000001e-06 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979169+0000", 2024-04-23T21:22:20.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7159999999999997e-06 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979175+0000", 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3360000000000002e-06 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98447 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984505+0000", 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037660846999999997, 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040997032000000003, 2024-04-23T21:22:20.232 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98447 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984505+0000", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984505+0000", 2024-04-23T21:22:20.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984505+0000", 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5400000000000002e-07 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984507+0000", 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2479999999999999e-06 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984513+0000", 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9710000000000001e-06 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984518+0000", 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6240000000000001e-06 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98398 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979181+0000", 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042984176999999998, 2024-04-23T21:22:20.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046349026000000002, 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98398 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979181+0000", 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979181+0000", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979182+0000", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979184+0000", 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.063e-06 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979190+0000", 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4370000000000004e-06 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979195+0000", 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9930000000000003e-06 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98449 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.238 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984548+0000", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037617802999999998, 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041009913000000002, 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98449 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984548+0000", 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984548+0000", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984548+0000", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984551+0000", 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.756e-06 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984558+0000", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.503e-06 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984564+0000", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3279999999999996e-06 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98400 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979201+0000", 2024-04-23T21:22:20.241 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042964255, 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046383855000000002, 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98400 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979201+0000", 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979201+0000", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979202+0000", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979204+0000", 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979210+0000", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979215+0000", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7910000000000004e-06 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98452 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.984571+0000", 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.037594206999999998, 2024-04-23T21:22:20.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.041075885999999999, 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98452 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984571+0000", 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984571+0000", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984572+0000", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984574+0000", 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.058e-06 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984578+0000", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000004e-06 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.984580+0000", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6010000000000002e-06 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99227 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.002038+0000", 2024-04-23T21:22:20.247 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.020127453999999999, 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023637154, 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99227 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002038+0000", 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002038+0000", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002039+0000", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2240000000000001e-06 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002045+0000", 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002058+0000", 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.271e-05 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002066+0000", 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0609999999999996e-06 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98402 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.250 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979221+0000", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042944384000000002, 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046481682000000003, 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98402 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979221+0000", 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979221+0000", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979222+0000", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979224+0000", 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0949999999999998e-06 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979230+0000", 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5860000000000001e-06 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979235+0000", 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0860000000000001e-06 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99228 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.253 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.002469+0000", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019696168999999999, 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023260794000000001, 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99228 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002469+0000", 2024-04-23T21:22:20.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002469+0000", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002470+0000", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6799999999999999e-07 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002475+0000", 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:20.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002488+0000", 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2787e-05 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002496+0000", 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8609999999999998e-06 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98404 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.256 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979403+0000", 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042762347999999999, 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046354748000000001, 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98404 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979403+0000", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979403+0000", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979404+0000", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8400000000000004e-07 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979417+0000", 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2815e-05 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979425+0000", 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2640000000000008e-06 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979431+0000", 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3559999999999998e-06 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98454 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.985973+0000", 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.036192202, 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039814422000000002, 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98454 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985973+0000", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985973+0000", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985974+0000", 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4099999999999999e-07 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985977+0000", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.162e-06 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985984+0000", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3490000000000003e-06 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.985991+0000", 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.973e-06 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.262 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99230 3.1d 3.e13f695d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.002508+0000", 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019657899, 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023311325000000001, 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.263 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99230 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002508+0000", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002508+0000", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002508+0000", 2024-04-23T21:22:20.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1999999999999999e-07 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002511+0000", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8109999999999999e-06 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002521+0000", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5289999999999997e-06 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002528+0000", 2024-04-23T21:22:20.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8150000000000006e-06 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98406 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979442+0000", 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042723219, 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046404256999999997, 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.266 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98406 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979442+0000", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979442+0000", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979443+0000", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4700000000000001e-07 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979445+0000", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979452+0000", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9349999999999997e-06 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979458+0000", 2024-04-23T21:22:20.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3109999999999996e-06 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98408 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979464+0000", 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042701489000000002, 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046410167000000002, 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.269 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98408 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979464+0000", 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979464+0000", 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979465+0000", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979467+0000", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0669999999999999e-06 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979473+0000", 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5830000000000003e-06 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979478+0000", 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.967e-06 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98470 3.5 3.5fb365 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.986022+0000", 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.036143578000000003, 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039881565000000001, 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.272 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98470 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986022+0000", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986022+0000", 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986022+0000", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8699999999999995e-07 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986025+0000", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2929999999999999e-06 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986031+0000", 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986037+0000", 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98410 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979485+0000", 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042681011999999997, 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046461032999999999, 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.275 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98410 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979485+0000", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979485+0000", 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979485+0000", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4000000000000002e-07 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979487+0000", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.013e-06 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979494+0000", 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4880000000000004e-06 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979498+0000", 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9479999999999999e-06 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98472 3.5 3.5fb365 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.986044+0000", 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.036122003999999999, 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039935026999999998, 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.278 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98472 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986044+0000", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986044+0000", 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986044+0000", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986046+0000", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.114e-06 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986052+0000", 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3659999999999997e-06 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986058+0000", 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1109999999999999e-06 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99250 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.002588+0000", 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019577924, 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023418660000000001, 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.281 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99250 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002588+0000", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002588+0000", 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002588+0000", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6600000000000002e-07 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002591+0000", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7369999999999998e-06 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002598+0000", 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1520000000000003e-06 2024-04-23T21:22:20.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002603+0000", 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98412 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979505+0000", 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042660511999999998, 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046539824, 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.284 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98412 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979505+0000", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979505+0000", 2024-04-23T21:22:20.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979505+0000", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.08e-07 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979508+0000", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.165e-06 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979515+0000", 2024-04-23T21:22:20.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9430000000000003e-06 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979520+0000", 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0259999999999998e-06 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98474 3.5 3.5fb365 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.986064+0000", 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.036101728999999999, 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.040033522000000002, 2024-04-23T21:22:20.287 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98474 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986064+0000", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986064+0000", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986064+0000", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986071+0000", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4829999999999996e-06 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986078+0000", 2024-04-23T21:22:20.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8449999999999997e-06 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.986083+0000", 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3419999999999998e-06 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99251 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.002618+0000", 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019547934999999999, 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023509358000000001, 2024-04-23T21:22:20.290 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99251 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002618+0000", 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002618+0000", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002618+0000", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8500000000000002e-07 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002621+0000", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5179999999999999e-06 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002628+0000", 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1779999999999997e-06 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.002633+0000", 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4399999999999996e-06 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98414 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979526+0000", 2024-04-23T21:22:20.293 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042639165, 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046629177000000001, 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98414 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979526+0000", 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979526+0000", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979649+0000", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000122602 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979653+0000", 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.647e-06 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979662+0000", 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8699999999999998e-06 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979668+0000", 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99280 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.296 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.003210+0000", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018955171, 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022999687000000001, 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99280 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003210+0000", 2024-04-23T21:22:20.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003210+0000", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003211+0000", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9400000000000005e-07 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003215+0000", 2024-04-23T21:22:20.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1649999999999999e-06 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003224+0000", 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5499999999999995e-06 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003231+0000", 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3900000000000004e-06 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.299 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:98415 3.1d 3.f9ccd89d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.979678+0000", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.042487358000000003, 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.046586225000000002, 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 98415 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979678+0000", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.979678+0000", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980932+0000", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012540699999999999 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980937+0000", 2024-04-23T21:22:20.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5190000000000003e-06 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980960+0000", 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2929999999999999e-05 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.980968+0000", 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999992e-06 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:99281 3.5 3.fdc2c785 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:20.003999+0000", 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018166257000000002, 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022319813000000001, 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 99281 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003999+0000", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.003999+0000", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.004000+0000", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5099999999999999e-07 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.004004+0000", 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000004e-06 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.004030+0000", 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5873000000000001e-05 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.004038+0000", 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7989999999999994e-06 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.305 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.306 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:20.306 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 88 2024-04-23T21:22:20.306 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:20.307 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:22:20.307 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T21:22:20.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:20.309 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-23T21:22:20.310 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:20.307+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:20.408 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:20.409+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:20.501 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:20.502 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:20.502 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:20.502 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:20.502 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:20.509 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:20.510+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:20.514 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:20.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:20.610 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:20.611+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:20.667 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.79167451600000005, 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:20.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72525600700000004, 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:20.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.725238626, 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:20.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72522106900000005, 2024-04-23T21:22:20.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72356087400000002, 2024-04-23T21:22:20.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:20.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.723538978, 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72351162800000002, 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:20.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:20.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:20.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:20.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72348754000000004, 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.698 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72345648100000004, 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.702 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:20.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:20.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:20.706 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72225641299999999, 2024-04-23T21:22:20.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:20.709 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.709 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.709 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:20.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:20.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:20.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72223398000000005, 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72221039200000003, 2024-04-23T21:22:20.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:20.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:20.721 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72219045299999995, 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:20.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:20.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:20.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72216197800000004, 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.72077923399999999, 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.71568261, 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:20.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:20.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:20.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.71533598399999998, 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.738 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:20.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:20.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.70382107999999999, 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.70380114100000002, 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.70250087000000005, 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:20.750 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:20.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:20.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:20.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:20.754 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:20.755 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:20.756 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:20.756 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:20.756 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:20.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:20.757 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:20.757 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:20.758 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:20.755+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:20.761 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:20.762 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:20.779 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:20.779 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:20.796 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:20.796 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:20.814 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:20.814 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:20.831 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:20.831 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:20.883 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:20.883+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:20.885 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-04-23T21:22:20.905 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:20.905 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:20.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:20.957 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:20.957+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:21.027 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.027 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.028 INFO:teuthology.orchestra.run.smithi111.stdout:4564 write 0x842b3a5 thru 0x843a654 (0xf2b0 bytes) 2024-04-23T21:22:21.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:21.049 INFO:teuthology.orchestra.run.smithi111.stdout:4565 punch from 0x3bd64a1 to 0x3be17e3, (0xb342 bytes) 2024-04-23T21:22:21.049 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.051 INFO:teuthology.orchestra.run.smithi111.stdout:4567 trunc from 0xdc083ed to 0xc424e66 2024-04-23T21:22:21.058 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:21.058+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:21.077 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.078 INFO:teuthology.orchestra.run.smithi111.stdout:4569 write 0x2b02e4d thru 0x2b0f99f (0xcb53 bytes) 2024-04-23T21:22:21.124 INFO:teuthology.orchestra.run.smithi111.stdout:4570 write 0xe08f14f thru 0xe09a5c7 (0xb479 bytes) 2024-04-23T21:22:21.132 INFO:teuthology.orchestra.run.smithi111.stdout:4571 write 0x730f351 thru 0x731a5eb (0xb29b bytes) 2024-04-23T21:22:21.159 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:21.159+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:21.173 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.173 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.174 INFO:teuthology.orchestra.run.smithi111.stdout:4572 punch from 0x7867ffc to 0x786db9a, (0x5b9e bytes) 2024-04-23T21:22:21.175 INFO:teuthology.orchestra.run.smithi111.stdout:4573 write 0x10f9bd6 thru 0x1106c90 (0xd0bb bytes) 2024-04-23T21:22:21.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:21.202 INFO:teuthology.orchestra.run.smithi111.stdout:4575 write 0x6952304 thru 0x69621f3 (0xfef0 bytes) 2024-04-23T21:22:21.238 INFO:teuthology.orchestra.run.smithi111.stdout:4576 write 0xdd8c78c thru 0xdd8d657 (0xecc bytes) 2024-04-23T21:22:21.241 INFO:teuthology.orchestra.run.smithi111.stdout:4579 read 0x6cf4f93 thru 0x6cf63a4 (0x1412 bytes) 2024-04-23T21:22:21.241 INFO:teuthology.orchestra.run.smithi111.stdout:4581 read 0x332073f thru 0x3322938 (0x21fa bytes) 2024-04-23T21:22:21.242 INFO:teuthology.orchestra.run.smithi111.stdout:4582 punch from 0xa2e1cab to 0xa2f04c2, (0xe817 bytes) 2024-04-23T21:22:21.242 INFO:teuthology.orchestra.run.smithi111.stdout:4583 writesame 0x7970e87 thru 0x7977dd6 data_size 0x34(0x6f50 bytes) 2024-04-23T21:22:21.243 INFO:teuthology.orchestra.run.smithi111.stdout:4584 punch from 0x70f2468 to 0x70fc549, (0xa0e1 bytes) 2024-04-23T21:22:21.244 INFO:teuthology.orchestra.run.smithi111.stdout:4586 read 0x5de92b thru 0x5e0d87 (0x245d bytes) 2024-04-23T21:22:21.244 INFO:teuthology.orchestra.run.smithi111.stdout:4587 punch from 0xaf62376 to 0xaf6cb29, (0xa7b3 bytes) 2024-04-23T21:22:21.244 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.244 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.245 INFO:teuthology.orchestra.run.smithi111.stdout:4590 read 0x4daaf83 thru 0x4db6652 (0xb6d0 bytes) 2024-04-23T21:22:21.246 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.246 INFO:teuthology.orchestra.run.smithi111.stdout:4592 writesame 0x8deb5ce thru 0x8ded4d2 data_size 0x1(0x1f05 bytes) 2024-04-23T21:22:21.260 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:21.261+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:21.262 INFO:teuthology.orchestra.run.smithi111.stdout:4595 trunc from 0xe09a5c8 to 0x904612d 2024-04-23T21:22:21.279 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T21:22:21.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-23T21:22:21.324 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.324 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:21.339 INFO:teuthology.orchestra.run.smithi111.stdout:4597 trunc from 0x904612d to 0x86578a0 2024-04-23T21:22:21.381 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:21.381+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:21.422 INFO:teuthology.orchestra.run.smithi111.stdout:4598 read 0x3a425d9 thru 0x3a44370 (0x1d98 bytes) 2024-04-23T21:22:21.422 INFO:teuthology.orchestra.run.smithi111.stdout:4599 read 0x49c61eb thru 0x49c8a3e (0x2854 bytes) 2024-04-23T21:22:21.423 INFO:teuthology.orchestra.run.smithi111.stdout:4602 punch from 0x1b0a04f to 0x1b1354e, (0x94ff bytes) 2024-04-23T21:22:21.429 INFO:teuthology.orchestra.run.smithi111.stdout:4604 trunc from 0x86578a0 to 0x3c4de0e 2024-04-23T21:22:21.466 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:21.462+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:21.530 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.530 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:21.545 INFO:teuthology.orchestra.run.smithi111.stdout:4605 trunc from 0x3c4de0e to 0x58959f4 2024-04-23T21:22:21.551 INFO:teuthology.orchestra.run.smithi111.stdout:4606 write 0x3fb87e9 thru 0x3fbe1de (0x59f6 bytes) 2024-04-23T21:22:21.555 INFO:teuthology.orchestra.run.smithi111.stdout:4608 read 0x26e70d3 thru 0x26e7ca3 (0xbd1 bytes) 2024-04-23T21:22:21.562 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:21.562+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:21.565 INFO:teuthology.orchestra.run.smithi111.stdout:4609 write 0x92aeeeb thru 0x92b7e1c (0x8f32 bytes) 2024-04-23T21:22:21.578 INFO:teuthology.orchestra.run.smithi111.stdout:4610 trunc from 0x92b7e1d to 0x13e747f 2024-04-23T21:22:21.664 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:21.664+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:21.736 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.737 INFO:teuthology.orchestra.run.smithi111.stdout:4612 read 0x1033cd6 thru 0x1039cc4 (0x5fef bytes) 2024-04-23T21:22:21.737 INFO:teuthology.orchestra.run.smithi111.stdout:4613 write 0xfb68aa thru 0xfbfba4 (0x92fb bytes) 2024-04-23T21:22:21.746 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.747 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:21.759 INFO:teuthology.orchestra.run.smithi111.stdout:4614 write 0x6eb82a0 thru 0x6eb86f3 (0x454 bytes) 2024-04-23T21:22:21.761 INFO:teuthology.orchestra.run.smithi111.stdout:4619 read 0x3b6a437 thru 0x3b6d983 (0x354d bytes) 2024-04-23T21:22:21.761 INFO:teuthology.orchestra.run.smithi111.stdout:4620 read 0x1a4e565 thru 0x1a5cead (0xe949 bytes) 2024-04-23T21:22:21.768 INFO:teuthology.orchestra.run.smithi111.stdout:4621 writesame 0x8dbd992 thru 0x8dbfb3d data_size 0xa(0x21ac bytes) 2024-04-23T21:22:21.774 INFO:teuthology.orchestra.run.smithi111.stdout:4623 trunc from 0x8dbfb3e to 0xd247c0a 2024-04-23T21:22:21.781 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:21.781 INFO:teuthology.orchestra.run.smithi111.stdout:4627 read 0x8f53bb8 thru 0x8f5bd6a (0x81b3 bytes) 2024-04-23T21:22:21.781 INFO:teuthology.orchestra.run.smithi111.stdout:4628 write 0xa27e93d thru 0xa27fe50 (0x1514 bytes) 2024-04-23T21:22:21.792 INFO:teuthology.orchestra.run.smithi111.stdout:4629 trunc from 0xd247c0a to 0x6b1f452 2024-04-23T21:22:21.797 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:21.797+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:21.810 INFO:teuthology.orchestra.run.smithi111.stdout:4631 read 0x4b4607f thru 0x4b4d52c (0x74ae bytes) 2024-04-23T21:22:21.811 INFO:teuthology.orchestra.run.smithi111.stdout:4632 writesame 0x265cc5e thru 0x2667522 data_size 0x1(0xa8c5 bytes) 2024-04-23T21:22:21.842 INFO:teuthology.orchestra.run.smithi111.stdout:4633 writesame 0xa39bff6 thru 0xa3a3164 data_size 0x1(0x716f bytes) 2024-04-23T21:22:21.853 INFO:teuthology.orchestra.run.smithi111.stdout:4634 writesame 0xd37ddbe thru 0xd386a02 data_size 0x1(0x8c45 bytes) 2024-04-23T21:22:21.865 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:21.865+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:21.867 INFO:teuthology.orchestra.run.smithi111.stdout:4636 trunc from 0xd386a03 to 0xadb4f7 2024-04-23T21:22:21.869 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:21.869 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:21.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:21.965 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:21.966+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:22.048 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.048 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.058 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:22.066 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:22.067+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:22.089 INFO:teuthology.orchestra.run.smithi111.stdout:4637 read 0x83a005 thru 0x848f2f (0xef2b bytes) 2024-04-23T21:22:22.103 INFO:teuthology.orchestra.run.smithi111.stdout:4639 trunc from 0xadb4f7 to 0xb6dff71 2024-04-23T21:22:22.108 INFO:teuthology.orchestra.run.smithi111.stdout:4640 write 0x17baad7 thru 0x17bb9c0 (0xeea bytes) 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:4643 read 0x66f3646 thru 0x67028a8 (0xf263 bytes) 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:4644 read 0x91bbc94 thru 0x91cb34f (0xf6bc bytes) 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.112 INFO:teuthology.orchestra.run.smithi111.stdout:4649 write 0x935a98c thru 0x935ec34 (0x42a9 bytes) 2024-04-23T21:22:22.115 INFO:teuthology.orchestra.run.smithi111.stdout:4650 write 0x3a03646 thru 0x3a05543 (0x1efe bytes) 2024-04-23T21:22:22.118 INFO:teuthology.orchestra.run.smithi111.stdout:4651 read 0xb2edc65 thru 0xb2f2050 (0x43ec bytes) 2024-04-23T21:22:22.118 INFO:teuthology.orchestra.run.smithi111.stdout:4652 punch from 0x19122fa to 0x191d536, (0xb23c bytes) 2024-04-23T21:22:22.118 INFO:teuthology.orchestra.run.smithi111.stdout:4654 write 0x1cc367f thru 0x1ccf8a9 (0xc22b bytes) 2024-04-23T21:22:22.122 INFO:teuthology.orchestra.run.smithi111.stdout:4656 write 0x44e1f3c thru 0x44eb34c (0x9411 bytes) 2024-04-23T21:22:22.138 INFO:teuthology.orchestra.run.smithi111.stdout:4658 write 0xeaf13b3 thru 0xeaf74b2 (0x6100 bytes) 2024-04-23T21:22:22.151 INFO:teuthology.orchestra.run.smithi111.stdout:4659 trunc from 0xeaf74b3 to 0x4d8a1da 2024-04-23T21:22:22.161 INFO:teuthology.orchestra.run.smithi111.stdout:4660 read 0x7356b8 thru 0x74560a (0xff53 bytes) 2024-04-23T21:22:22.167 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:22.168+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:22.170 INFO:teuthology.orchestra.run.smithi111.stdout:4661 write 0x7a85f94 thru 0x7a8eb6b (0x8bd8 bytes) 2024-04-23T21:22:22.172 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.172 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.174 INFO:teuthology.orchestra.run.smithi111.stdout:4662 write 0x1b326d3 thru 0x1b3f8e7 (0xd215 bytes) 2024-04-23T21:22:22.182 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:22.185 INFO:teuthology.orchestra.run.smithi111.stdout:4664 trunc from 0x7a8eb6c to 0x28a8684 2024-04-23T21:22:22.214 INFO:teuthology.orchestra.run.smithi111.stdout:4665 write 0x4cbbbd7 thru 0x4cc3ddc (0x8206 bytes) 2024-04-23T21:22:22.232 INFO:teuthology.orchestra.run.smithi111.stdout:4666 writesame 0x985eef2 thru 0x98629c4 data_size 0xb(0x3ad3 bytes) 2024-04-23T21:22:22.235 INFO:teuthology.orchestra.run.smithi111.stdout:4667 trunc from 0x98629c5 to 0xb392806 2024-04-23T21:22:22.246 INFO:teuthology.orchestra.run.smithi111.stdout:4668 writesame 0xb6f2a7b thru 0xb6f8103 data_size 0x1(0x5689 bytes) 2024-04-23T21:22:22.246 INFO:teuthology.orchestra.run.smithi111.stdout:4669 read 0x9cc9c17 thru 0x9cd97e3 (0xfbcd bytes) 2024-04-23T21:22:22.246 INFO:teuthology.orchestra.run.smithi111.stdout:4670 write 0x6d463d0 thru 0x6d51fcd (0xbbfe bytes) 2024-04-23T21:22:22.253 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.253 INFO:teuthology.orchestra.run.smithi111.stdout:4673 punch from 0x36a7a50 to 0x36b01ae, (0x875e bytes) 2024-04-23T21:22:22.260 INFO:teuthology.orchestra.run.smithi111.stdout:4674 trunc from 0xb6f8104 to 0x611ee6d 2024-04-23T21:22:22.268 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:22.268+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:22.271 INFO:teuthology.orchestra.run.smithi111.stdout:4676 punch from 0x4dabff6 to 0x4daf2e6, (0x32f0 bytes) 2024-04-23T21:22:22.271 INFO:teuthology.orchestra.run.smithi111.stdout:4678 read 0x25fe87 thru 0x26f51d (0xf697 bytes) 2024-04-23T21:22:22.272 INFO:teuthology.orchestra.run.smithi111.stdout:4679 read 0x3196ab1 thru 0x319cf21 (0x6471 bytes) 2024-04-23T21:22:22.272 INFO:teuthology.orchestra.run.smithi111.stdout:4680 write 0x42ebf17 thru 0x42f6c36 (0xad20 bytes) 2024-04-23T21:22:22.277 INFO:teuthology.orchestra.run.smithi111.stdout:4681 write 0x2056458 thru 0x2062d5d (0xc906 bytes) 2024-04-23T21:22:22.286 INFO:teuthology.orchestra.run.smithi111.stdout:4685 punch from 0x2e877ae to 0x2e8a3e4, (0x2c36 bytes) 2024-04-23T21:22:22.288 INFO:teuthology.orchestra.run.smithi111.stdout:4686 trunc from 0x611ee6d to 0x4827871 2024-04-23T21:22:22.289 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T21:22:22.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-23T21:22:22.310 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:22:22.317 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.317 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.324 INFO:teuthology.orchestra.run.smithi111.stdout:4687 write 0x9d575ab thru 0x9d63ec0 (0xc916 bytes) 2024-04-23T21:22:22.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:22.330 INFO:teuthology.orchestra.run.smithi111.stdout:4689 punch from 0x6d2dcab to 0x6d329be, (0x4d13 bytes) 2024-04-23T21:22:22.330 INFO:teuthology.orchestra.run.smithi111.stdout:4690 read 0x4ea09bf thru 0x4ea5ddf (0x5421 bytes) 2024-04-23T21:22:22.330 INFO:teuthology.orchestra.run.smithi111.stdout:4691 writesame 0x6227492 thru 0x622f02c data_size 0x1(0x7b9b bytes) 2024-04-23T21:22:22.345 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.345 INFO:teuthology.orchestra.run.smithi111.stdout:4693 punch from 0x388069d to 0x388c5cd, (0xbf30 bytes) 2024-04-23T21:22:22.357 INFO:teuthology.orchestra.run.smithi111.stdout:4697 trunc from 0x9d63ec1 to 0xfe17c6 2024-04-23T21:22:22.369 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:22.370+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:22.371 INFO:teuthology.orchestra.run.smithi111.stdout:4699 read 0x2bd8d4 thru 0x2c0d54 (0x3481 bytes) 2024-04-23T21:22:22.371 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.386 INFO:teuthology.orchestra.run.smithi111.stdout:4702 trunc from 0xfe17c6 to 0xc5f720c 2024-04-23T21:22:22.392 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.400 INFO:teuthology.orchestra.run.smithi111.stdout:4704 trunc from 0xc5f720c to 0x5dafda6 2024-04-23T21:22:22.411 INFO:teuthology.orchestra.run.smithi111.stdout:4706 trunc from 0x5dafda6 to 0x9e42ba4 2024-04-23T21:22:22.416 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.422 INFO:teuthology.orchestra.run.smithi111.stdout:4708 writesame 0xa2ed2d8 thru 0xa2f590a data_size 0x1(0x8633 bytes) 2024-04-23T21:22:22.422 INFO:teuthology.orchestra.run.smithi111.stdout:4709 punch from 0x2d6caae to 0x2d7a6f2, (0xdc44 bytes) 2024-04-23T21:22:22.430 INFO:teuthology.orchestra.run.smithi111.stdout:4710 write 0xc23375b thru 0xc23b56f (0x7e15 bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4712 punch from 0x44f1b4 to 0x45e98c, (0xf7d8 bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4716 punch from 0x663ebd1 to 0x66464f7, (0x7926 bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4717 read 0x1f94d78 thru 0x1f95841 (0xaca bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4718 read 0x9281e44 thru 0x928a85b (0x8a18 bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4719 punch from 0x5138ba8 to 0x513915a, (0x5b2 bytes) 2024-04-23T21:22:22.435 INFO:teuthology.orchestra.run.smithi111.stdout:4720 write 0xca7302 thru 0xcae659 (0x7358 bytes) 2024-04-23T21:22:22.442 INFO:teuthology.orchestra.run.smithi111.stdout:4721 write 0xcddc622 thru 0xcdde04b (0x1a2a bytes) 2024-04-23T21:22:22.446 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.453 INFO:teuthology.orchestra.run.smithi111.stdout:4724 writesame 0xdc68112 thru 0xdc71281 data_size 0xd(0x9170 bytes) 2024-04-23T21:22:22.453 INFO:teuthology.orchestra.run.smithi111.stdout:4725 read 0xa8f3e0b thru 0xa8f4127 (0x31d bytes) 2024-04-23T21:22:22.454 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.454 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.457 INFO:teuthology.orchestra.run.smithi111.stdout:4726 trunc from 0xdc71282 to 0xae0bf66 2024-04-23T21:22:22.467 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:22.467 INFO:teuthology.orchestra.run.smithi111.stdout:4727 punch from 0x942bb48 to 0x94374dd, (0xb995 bytes) 2024-04-23T21:22:22.467 INFO:teuthology.orchestra.run.smithi111.stdout:4728 write 0x3275abc thru 0x3279007 (0x354c bytes) 2024-04-23T21:22:22.469 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:22.470+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:22.473 INFO:teuthology.orchestra.run.smithi111.stdout:4729 trunc from 0xae0bf66 to 0xd2ac4d2 2024-04-23T21:22:22.480 INFO:teuthology.orchestra.run.smithi111.stdout:4730 read 0x1cc4f62 thru 0x1cc5794 (0x833 bytes) 2024-04-23T21:22:22.480 INFO:teuthology.orchestra.run.smithi111.stdout:4731 read 0xb506eb7 thru 0xb5127b1 (0xb8fb bytes) 2024-04-23T21:22:22.481 INFO:teuthology.orchestra.run.smithi111.stdout:4732 punch from 0x23980d3 to 0x239c013, (0x3f40 bytes) 2024-04-23T21:22:22.487 INFO:teuthology.orchestra.run.smithi111.stdout:4733 write 0xdba88e7 thru 0xdbb06ec (0x7e06 bytes) 2024-04-23T21:22:22.494 INFO:teuthology.orchestra.run.smithi111.stdout:4734 write 0x73bb40b thru 0x73c1112 (0x5d08 bytes) 2024-04-23T21:22:22.500 INFO:teuthology.orchestra.run.smithi111.stdout:4735 read 0xc5f2a9 thru 0xc646db (0x5433 bytes) 2024-04-23T21:22:22.500 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.500 INFO:teuthology.orchestra.run.smithi111.stdout:4739 punch from 0x481872 to 0x48e084, (0xc812 bytes) 2024-04-23T21:22:22.500 INFO:teuthology.orchestra.run.smithi111.stdout:4742 read 0x95ae100 thru 0x95b1eef (0x3df0 bytes) 2024-04-23T21:22:22.501 INFO:teuthology.orchestra.run.smithi111.stdout:4743 punch from 0x5064f2a to 0x5071e9c, (0xcf72 bytes) 2024-04-23T21:22:22.501 INFO:teuthology.orchestra.run.smithi111.stdout:4744 read 0xbaedb31 thru 0xbaf4b63 (0x7033 bytes) 2024-04-23T21:22:22.501 INFO:teuthology.orchestra.run.smithi111.stdout:4746 write 0xa152b60 thru 0xa1576eb (0x4b8c bytes) 2024-04-23T21:22:22.504 INFO:teuthology.orchestra.run.smithi111.stdout:4747 read 0x99f15e1 thru 0x99fd74e (0xc16e bytes) 2024-04-23T21:22:22.504 INFO:teuthology.orchestra.run.smithi111.stdout:4748 write 0x1c7b238 thru 0x1c80883 (0x564c bytes) 2024-04-23T21:22:22.511 INFO:teuthology.orchestra.run.smithi111.stdout:4751 read 0x7c5f527 thru 0x7c66568 (0x7042 bytes) 2024-04-23T21:22:22.511 INFO:teuthology.orchestra.run.smithi111.stdout:4752 read 0x3fdb7bc thru 0x3fdd0ae (0x18f3 bytes) 2024-04-23T21:22:22.526 INFO:teuthology.orchestra.run.smithi111.stdout:4753 trunc from 0xdbb06ed to 0x34b418f 2024-04-23T21:22:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:4757 read 0xe75aa3 thru 0xe83fbb (0xe519 bytes) 2024-04-23T21:22:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:4759 punch from 0x31d0b15 to 0x31d77df, (0x6cca bytes) 2024-04-23T21:22:22.542 INFO:teuthology.orchestra.run.smithi111.stdout:4761 punch from 0x2450792 to 0x2450e7c, (0x6ea bytes) 2024-04-23T21:22:22.545 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.545 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.559 INFO:teuthology.orchestra.run.smithi111.stdout:4764 write 0xba87d40 thru 0xba9199e (0x9c5f bytes) 2024-04-23T21:22:22.571 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:22.571+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:22.572 INFO:teuthology.orchestra.run.smithi111.stdout:4765 trunc from 0xba9199f to 0x77d1577 2024-04-23T21:22:22.584 INFO:teuthology.orchestra.run.smithi111.stdout:4766 writesame 0x76f2e79 thru 0x76fc79e data_size 0x1(0x9926 bytes) 2024-04-23T21:22:22.589 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.589 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.598 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:22.606 INFO:teuthology.orchestra.run.smithi111.stdout:4767 writesame 0x7e65508 thru 0x7e65f75 data_size 0xf(0xa6e bytes) 2024-04-23T21:22:22.607 INFO:teuthology.orchestra.run.smithi111.stdout:4768 write 0x28340b9 thru 0x28359b1 (0x18f9 bytes) 2024-04-23T21:22:22.609 INFO:teuthology.orchestra.run.smithi111.stdout:4769 punch from 0x2f87f99 to 0x2f8e2de, (0x6345 bytes) 2024-04-23T21:22:22.609 INFO:teuthology.orchestra.run.smithi111.stdout:4770 read 0x31048c2 thru 0x3106ded (0x252c bytes) 2024-04-23T21:22:22.621 INFO:teuthology.orchestra.run.smithi111.stdout:4771 write 0xcd5f6d3 thru 0xcd65408 (0x5d36 bytes) 2024-04-23T21:22:22.625 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.638 INFO:teuthology.orchestra.run.smithi111.stdout:4774 trunc from 0xcd65409 to 0x21a3397 2024-04-23T21:22:22.652 INFO:teuthology.orchestra.run.smithi111.stdout:4775 read 0x1458c05 thru 0x145d2d7 (0x46d3 bytes) 2024-04-23T21:22:22.652 INFO:teuthology.orchestra.run.smithi111.stdout:4777 read 0x4866a thru 0x50e25 (0x87bc bytes) 2024-04-23T21:22:22.664 INFO:teuthology.orchestra.run.smithi111.stdout:4778 writesame 0x59c930c thru 0x59ce1f7 data_size 0x2(0x4eec bytes) 2024-04-23T21:22:22.664 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.664 INFO:teuthology.orchestra.run.smithi111.stdout:4781 punch from 0x4410168 to 0x441a978, (0xa810 bytes) 2024-04-23T21:22:22.667 INFO:teuthology.orchestra.run.smithi111.stdout:4783 trunc from 0x59ce1f8 to 0x2e40e9b 2024-04-23T21:22:22.671 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:22.672+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:22.673 INFO:teuthology.orchestra.run.smithi111.stdout:4785 read 0x21befb8 thru 0x21ce353 (0xf39c bytes) 2024-04-23T21:22:22.677 INFO:teuthology.orchestra.run.smithi111.stdout:4786 trunc from 0x2e40e9b to 0xdaa28 2024-04-23T21:22:22.698 INFO:teuthology.orchestra.run.smithi111.stdout:4787 write 0xd81723 thru 0xd8d6bf (0xbf9d bytes) 2024-04-23T21:22:22.716 INFO:teuthology.orchestra.run.smithi111.stdout:4788 write 0x9bf0566 thru 0x9bf87c9 (0x8264 bytes) 2024-04-23T21:22:22.725 INFO:teuthology.orchestra.run.smithi111.stdout:4789 write 0xa3f4c7c thru 0xa40093d (0xbcc2 bytes) 2024-04-23T21:22:22.728 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.728 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.730 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.731 INFO:teuthology.orchestra.run.smithi111.stdout:4791 read 0x3bd5d47 thru 0x3bd7eb4 (0x216e bytes) 2024-04-23T21:22:22.731 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.731 INFO:teuthology.orchestra.run.smithi111.stdout:4793 write 0x34933b1 thru 0x34989a9 (0x55f9 bytes) 2024-04-23T21:22:22.736 INFO:teuthology.orchestra.run.smithi111.stdout:4794 write 0x2a1e45 thru 0x2adb6b (0xbd27 bytes) 2024-04-23T21:22:22.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:22.738 INFO:teuthology.orchestra.run.smithi111.stdout:4795 write 0x6d430a8 thru 0x6d4bb31 (0x8a8a bytes) 2024-04-23T21:22:22.744 INFO:teuthology.orchestra.run.smithi111.stdout:4797 write 0x9c8fc00 thru 0x9c8fe5e (0x25f bytes) 2024-04-23T21:22:22.747 INFO:teuthology.orchestra.run.smithi111.stdout:4798 read 0x78f9c27 thru 0x78fb02b (0x1405 bytes) 2024-04-23T21:22:22.749 INFO:teuthology.orchestra.run.smithi111.stdout:4799 trunc from 0xa40093e to 0xab111de 2024-04-23T21:22:22.755 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.760 INFO:teuthology.orchestra.run.smithi111.stdout:4801 trunc from 0xab111de to 0x6bae08c 2024-04-23T21:22:22.772 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:22.772+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:22.775 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.776 INFO:teuthology.orchestra.run.smithi111.stdout:4803 punch from 0x5ca59d8 to 0x5cabf45, (0x656d bytes) 2024-04-23T21:22:22.776 INFO:teuthology.orchestra.run.smithi111.stdout:4804 punch from 0x3ac2f66 to 0x3ad201a, (0xf0b4 bytes) 2024-04-23T21:22:22.776 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.785 INFO:teuthology.orchestra.run.smithi111.stdout:4806 writesame 0x9a6bc89 thru 0x9a6da4e data_size 0x1(0x1dc6 bytes) 2024-04-23T21:22:22.785 INFO:teuthology.orchestra.run.smithi111.stdout:4808 read 0x352e46f thru 0x3532444 (0x3fd6 bytes) 2024-04-23T21:22:22.785 INFO:teuthology.orchestra.run.smithi111.stdout:4809 writesame 0x3235a24 thru 0x324260e data_size 0x1(0xcbeb bytes) 2024-04-23T21:22:22.800 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.800 INFO:teuthology.orchestra.run.smithi111.stdout:4812 read 0x36019de thru 0x3604527 (0x2b4a bytes) 2024-04-23T21:22:22.801 INFO:teuthology.orchestra.run.smithi111.stdout:4813 read 0x64dbd6f thru 0x64e949b (0xd72d bytes) 2024-04-23T21:22:22.801 INFO:teuthology.orchestra.run.smithi111.stdout:4814 trunc from 0x9a6da4f to 0xa317124 2024-04-23T21:22:22.813 INFO:teuthology.orchestra.run.smithi111.stdout:4815 trunc from 0xa317124 to 0x613e5eb 2024-04-23T21:22:22.825 INFO:teuthology.orchestra.run.smithi111.stdout:4816 punch from 0x2634b34 to 0x263e41d, (0x98e9 bytes) 2024-04-23T21:22:22.826 INFO:teuthology.orchestra.run.smithi111.stdout:4817 read 0x4df7db5 thru 0x4e06c22 (0xee6e bytes) 2024-04-23T21:22:22.826 INFO:teuthology.orchestra.run.smithi111.stdout:4819 read 0xa4f2ad thru 0xa5976b (0xa4bf bytes) 2024-04-23T21:22:22.826 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.826 INFO:teuthology.orchestra.run.smithi111.stdout:4822 read 0x54ecb7c thru 0x54f9868 (0xcced bytes) 2024-04-23T21:22:22.826 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.827 INFO:teuthology.orchestra.run.smithi111.stdout:4825 punch from 0x3c39587 to 0x3c42cb7, (0x9730 bytes) 2024-04-23T21:22:22.827 INFO:teuthology.orchestra.run.smithi111.stdout:4827 punch from 0x16f753c to 0x1702620, (0xb0e4 bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4829 writesame 0x9d49346 thru 0x9d52a2c data_size 0x3(0x96e7 bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4831 punch from 0x7cc6c2b to 0x7cccf64, (0x6339 bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4832 read 0x23edd62 thru 0x23f1dbf (0x405e bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4834 read 0x1bf21ce thru 0x1bfabf6 (0x8a29 bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4835 read 0x5d597b7 thru 0x5d61c28 (0x8472 bytes) 2024-04-23T21:22:22.837 INFO:teuthology.orchestra.run.smithi111.stdout:4836 writesame 0x65faecf thru 0x6606487 data_size 0x9(0xb5b9 bytes) 2024-04-23T21:22:22.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:22.842 INFO:teuthology.orchestra.run.smithi111.stdout:4837 writesame 0x8af893e thru 0x8b066bc data_size 0x29(0xdd7f bytes) 2024-04-23T21:22:22.852 INFO:teuthology.orchestra.run.smithi111.stdout:4838 trunc from 0x9d52a2d to 0x476a9e6 2024-04-23T21:22:22.856 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.856 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.863 INFO:teuthology.orchestra.run.smithi111.stdout:4839 writesame 0x93b6c0 thru 0x947c03 data_size 0x19(0xc544 bytes) 2024-04-23T21:22:22.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:22.866 INFO:teuthology.orchestra.run.smithi111.stdout:4841 write 0x3168d94 thru 0x317641b (0xd688 bytes) 2024-04-23T21:22:22.873 INFO:teuthology.orchestra.run.smithi111.stdout:4842 read 0x44b3b4 thru 0x456e08 (0xba55 bytes) 2024-04-23T21:22:22.874 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:22.874+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:22.874 INFO:teuthology.orchestra.run.smithi111.stdout:4844 punch from 0x11e286c to 0x11f00b3, (0xd847 bytes) 2024-04-23T21:22:22.875 INFO:teuthology.orchestra.run.smithi111.stdout:4845 trunc from 0x476a9e6 to 0x56913d1 2024-04-23T21:22:22.898 INFO:teuthology.orchestra.run.smithi111.stdout:4846 writesame 0xecd672d thru 0xece254a data_size 0xa(0xbe1e bytes) 2024-04-23T21:22:22.898 INFO:teuthology.orchestra.run.smithi111.stdout:4847 write 0xffb488 thru 0xffe8ef (0x3468 bytes) 2024-04-23T21:22:22.901 INFO:teuthology.orchestra.run.smithi111.stdout:4848 read 0xf48761 thru 0xf48843 (0xe3 bytes) 2024-04-23T21:22:22.901 INFO:teuthology.orchestra.run.smithi111.stdout:4849 read 0xe4e3c2e thru 0xe4f16d7 (0xdaaa bytes) 2024-04-23T21:22:22.901 INFO:teuthology.orchestra.run.smithi111.stdout:4850 write 0x9f20021 thru 0x9f20da2 (0xd82 bytes) 2024-04-23T21:22:22.904 INFO:teuthology.orchestra.run.smithi111.stdout:4851 punch from 0x71b29e1 to 0x71b2c31, (0x250 bytes) 2024-04-23T21:22:22.904 INFO:teuthology.orchestra.run.smithi111.stdout:4852 writesame 0x6bbdd48 thru 0x6bc885c data_size 0xd(0xab15 bytes) 2024-04-23T21:22:22.911 INFO:teuthology.orchestra.run.smithi111.stdout:4853 trunc from 0xece254b to 0xde63518 2024-04-23T21:22:22.918 INFO:teuthology.orchestra.run.smithi111.stdout:4854 trunc from 0xde63518 to 0xd8928ac 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4855 read 0x9674f3b thru 0x9675080 (0x146 bytes) 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4858 read 0xaeed3d2 thru 0xaeedc16 (0x845 bytes) 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4859 punch from 0x8d2f408 to 0x8d37994, (0x858c bytes) 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4863 read 0x66a6d8f thru 0x66ad5db (0x684d bytes) 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4864 read 0x42fd8d1 thru 0x42fe8f8 (0x1028 bytes) 2024-04-23T21:22:22.924 INFO:teuthology.orchestra.run.smithi111.stdout:4865 write 0x9df9cdb thru 0x9e00e37 (0x715d bytes) 2024-04-23T21:22:22.927 INFO:teuthology.orchestra.run.smithi111.stdout:4866 write 0x3015413 thru 0x3024a89 (0xf677 bytes) 2024-04-23T21:22:22.940 INFO:teuthology.orchestra.run.smithi111.stdout:4867 writesame 0xe848034 thru 0xe84858f data_size 0xe(0x55c bytes) 2024-04-23T21:22:22.940 INFO:teuthology.orchestra.run.smithi111.stdout:4868 read 0x6b69089 thru 0x6b75978 (0xc8f0 bytes) 2024-04-23T21:22:22.946 INFO:teuthology.orchestra.run.smithi111.stdout:4870 writesame 0xebcd9d9 thru 0xebcf051 data_size 0x1(0x1679 bytes) 2024-04-23T21:22:22.946 INFO:teuthology.orchestra.run.smithi111.stdout:4873 punch from 0x83e2de3 to 0x83ec522, (0x973f bytes) 2024-04-23T21:22:22.946 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.946 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:22.946 INFO:teuthology.orchestra.run.smithi111.stdout:4876 clone 42 order 23 su 131072 sc 3 2024-04-23T21:22:22.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:22.971 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:22.973 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:22.974+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:22.982 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:22.982 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:22.988 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:22.989 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:22.991 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:23.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:23.006 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:23.022 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:23.023 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:23.041 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:23.041 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:23.058 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:23.059 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:23.074 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:23.074+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:23.075 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:23.076 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:23.093 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:23.093 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:23.116 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:23.116 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:23.116 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:23.116 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:23.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:23.209 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:23.209+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:23.267 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:23.267 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:23.267 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:23.267 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:23.267 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:23.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:23.276 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:23.276+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:23.296 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T21:22:23.304 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:22:23.313 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-23T21:22:23.376 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:23.376+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:23.405 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone41 from 0xebcf052 (overlap 0xdaa28) to 0x41972 2024-04-23T21:22:23.418 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:23.418 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:23.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:23.418 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.778395981000003, 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.419 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:23.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:23.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:23.422 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.778363874, 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:23.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:23.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:23.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.778340487999998, 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.432 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.778320180999998, 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:23.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.778300369, 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:23.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:23.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777228971, 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:23.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:23.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777195626000001, 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:23.445 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:23.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.449 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777164096999996, 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777136708, 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777124802000003, 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.458 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:23.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777106558, 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:23.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:23.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:23.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777096741000001, 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:23.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777086494000002, 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777076843000003, 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:23.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:23.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.777043343999999, 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:23.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:23.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.77341792, 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.482 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:23.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:23.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:23.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.773400051000003, 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:23.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:23.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.773381321999999, 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:23.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:23.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.773362894000002, 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.494 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:23.498 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.773344989000002, 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:23.502 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:23.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:23.504 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:23.504+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:23.539 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-04-23T21:22:23.604 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:23.605+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:23.680 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:23.680 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:23.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123828 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.642218+0000", 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019305119999999999, 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0193277, 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123828 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642218+0000", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642218+0000", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642218+0000", 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7999999999999995e-07 2024-04-23T21:22:23.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642221+0000", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5509999999999998e-06 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642228+0000", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7410000000000003e-06 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642233+0000", 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2780000000000001e-06 2024-04-23T21:22:23.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123948 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643104+0000", 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018419029999999999, 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018480766999999999, 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.684 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123948 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643104+0000", 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643104+0000", 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643104+0000", 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643107+0000", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4389999999999999e-06 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643113+0000", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5860000000000001e-06 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643118+0000", 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0660000000000003e-06 2024-04-23T21:22:23.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124059 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644103+0000", 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017419931999999999, 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017512249000000001, 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124059 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644103+0000", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644103+0000", 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644103+0000", 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3599999999999999e-07 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644106+0000", 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3999999999999999e-06 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644112+0000", 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.319e-06 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644117+0000", 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0250000000000002e-06 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124175 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644898+0000", 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016624366000000002, 2024-04-23T21:22:23.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016746492000000002, 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124175 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644898+0000", 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644898+0000", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644899+0000", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999998e-07 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644901+0000", 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.464e-06 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644908+0000", 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2489999999999998e-06 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644912+0000", 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8420000000000004e-06 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124274 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645774+0000", 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015749055000000001, 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015924130000000002, 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124274 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645774+0000", 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645774+0000", 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645774+0000", 2024-04-23T21:22:23.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645777+0000", 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.34e-06 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645783+0000", 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3040000000000001e-06 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645788+0000", 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8760000000000004e-06 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124404 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648873+0000", 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012650273, 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01288221, 2024-04-23T21:22:23.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124404 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648873+0000", 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648873+0000", 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648873+0000", 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648875+0000", 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3209999999999999e-06 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648882+0000", 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5379999999999999e-06 2024-04-23T21:22:23.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648887+0000", 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7829999999999998e-06 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.705 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124466 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651537+0000", 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0099860599999999997, 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010272732999999999, 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.706 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124466 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651537+0000", 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651537+0000", 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651537+0000", 2024-04-23T21:22:23.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651544+0000", 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2930000000000003e-06 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651551+0000", 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0380000000000002e-06 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651556+0000", 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.967e-06 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124600 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656187+0000", 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0053353009999999998, 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0056755419999999996, 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.710 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124600 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656187+0000", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656187+0000", 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656301+0000", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113485 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656305+0000", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6459999999999999e-06 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656314+0000", 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0559999999999994e-06 2024-04-23T21:22:23.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656322+0000", 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7180000000000002e-06 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123835 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.642305+0000", 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019218122000000001, 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019613614000000001, 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123835 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642305+0000", 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642305+0000", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642305+0000", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.37e-07 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642308+0000", 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9399999999999998e-06 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642316+0000", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0150000000000003e-06 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642322+0000", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9390000000000002e-06 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123949 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643125+0000", 2024-04-23T21:22:23.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018397833999999998, 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018856141, 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123949 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643125+0000", 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643125+0000", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643126+0000", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5700000000000002e-07 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643136+0000", 2024-04-23T21:22:23.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0883000000000001e-05 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643143+0000", 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9489999999999999e-06 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643149+0000", 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4480000000000002e-06 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124062 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644123+0000", 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017399562, 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017912741999999999, 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124062 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644123+0000", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644123+0000", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644124+0000", 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.96e-07 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644126+0000", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1900000000000002e-06 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644133+0000", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4619999999999999e-06 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644138+0000", 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0749999999999997e-06 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.722 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124182 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644919+0000", 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016604170000000001, 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017172768000000001, 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124182 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.723 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644919+0000", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644919+0000", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644919+0000", 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644922+0000", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5459999999999998e-06 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644935+0000", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3064999999999999e-05 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644940+0000", 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4360000000000001e-06 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124275 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645794+0000", 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015729061999999999, 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016352179000000001, 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124275 2024-04-23T21:22:23.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645794+0000", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645794+0000", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645794+0000", 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2699999999999999e-07 2024-04-23T21:22:23.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645803+0000", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2059999999999996e-06 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645809+0000", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9970000000000004e-06 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645814+0000", 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.045e-06 2024-04-23T21:22:23.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124344 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647769+0000", 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01375376, 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0144318, 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.729 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124344 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647769+0000", 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647769+0000", 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647769+0000", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647771+0000", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.813e-06 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647777+0000", 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6640000000000003e-06 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647781+0000", 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5859999999999998e-06 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124405 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648893+0000", 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012629712, 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013364050000000001, 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124405 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648893+0000", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648893+0000", 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648893+0000", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648895+0000", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9580000000000001e-06 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648901+0000", 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3669999999999996e-06 2024-04-23T21:22:23.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648905+0000", 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6240000000000001e-06 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124606 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656332+0000", 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0051903039999999998, 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0059839300000000002, 2024-04-23T21:22:23.735 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124606 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656332+0000", 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656332+0000", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656974+0000", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00064135899999999998 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656979+0000", 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.668e-06 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656988+0000", 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3880000000000006e-06 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656995+0000", 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7190000000000004e-06 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.738 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124669 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659420+0000", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002102685, 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.002950299, 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124669 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659420+0000", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659420+0000", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659421+0000", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.61e-07 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659423+0000", 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3649999999999998e-06 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659430+0000", 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5060000000000001e-06 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659435+0000", 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0499999999999999e-06 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.741 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123836 3.1 3:876dd341:::rbd_data.1118f9168b6.0000000000000008:29 [read 1269760~4096 out=4096b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.642339+0000", 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.019183983000000002, 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020097535, 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.742 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123836 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642339+0000", 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642339+0000", 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642339+0000", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5700000000000002e-07 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642342+0000", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5100000000000001e-06 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642349+0000", 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2219999999999996e-06 2024-04-23T21:22:23.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.642355+0000", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5400000000000003e-06 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.662286+0000", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019931401000000001 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.662363+0000", 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7199000000000004e-05 2024-04-23T21:22:23.745 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123952 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643158+0000", 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018365266000000002, 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019345503999999999, 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123952 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643158+0000", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643158+0000", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643158+0000", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1600000000000001e-07 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643160+0000", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2800000000000002e-06 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643181+0000", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0455e-05 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643188+0000", 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7549999999999997e-06 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124064 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644145+0000", 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017377825, 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018411528, 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124064 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644145+0000", 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644145+0000", 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644145+0000", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.77e-07 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644148+0000", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1730000000000002e-06 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644154+0000", 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0510000000000002e-06 2024-04-23T21:22:23.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644158+0000", 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6450000000000004e-06 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124183 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644947+0000", 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016575515999999998, 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017663099000000002, 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124183 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644947+0000", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644947+0000", 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644948+0000", 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644950+0000", 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2170000000000001e-06 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644956+0000", 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1360000000000002e-06 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644961+0000", 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9640000000000002e-06 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124345 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.755 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647788+0000", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013735165000000001, 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014879402, 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124345 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647788+0000", 2024-04-23T21:22:23.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647788+0000", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647788+0000", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647790+0000", 2024-04-23T21:22:23.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647795+0000", 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4820000000000002e-06 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647800+0000", 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5630000000000002e-06 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124412 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648911+0000", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012611693, 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013810309, 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124412 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648911+0000", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648911+0000", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648912+0000", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648914+0000", 2024-04-23T21:22:23.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.894e-06 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648919+0000", 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1549999999999998e-06 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648923+0000", 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4789999999999998e-06 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124537 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655014+0000", 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0065090199999999999, 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0077613250000000003, 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124537 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655014+0000", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655014+0000", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655014+0000", 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.65e-07 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655016+0000", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0140000000000001e-06 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655031+0000", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4595e-05 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655036+0000", 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0520000000000001e-06 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.764 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124608 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.657006+0000", 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0045162900000000001, 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0058224959999999999, 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124608 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.765 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657006+0000", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657006+0000", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657007+0000", 2024-04-23T21:22:23.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0200000000000003e-07 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657009+0000", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.249e-06 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657015+0000", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0820000000000004e-06 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657020+0000", 2024-04-23T21:22:23.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0499999999999999e-06 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124670 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659441+0000", 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002081327, 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.003442353, 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124670 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659441+0000", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659441+0000", 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659442+0000", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-07 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659444+0000", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9460000000000001e-06 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659450+0000", 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7100000000000004e-06 2024-04-23T21:22:23.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659455+0000", 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.101e-06 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.773 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123958 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.773 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643202+0000", 2024-04-23T21:22:23.773 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018320768000000001, 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019742114000000002, 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123958 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643202+0000", 2024-04-23T21:22:23.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643202+0000", 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643203+0000", 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3300000000000002e-07 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643205+0000", 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4389999999999999e-06 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643212+0000", 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6610000000000003e-06 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643217+0000", 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3469999999999998e-06 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124075 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644164+0000", 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017358424000000001, 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018834022999999998, 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124075 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644164+0000", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644164+0000", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644165+0000", 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644167+0000", 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1679999999999998e-06 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644173+0000", 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1330000000000004e-06 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644178+0000", 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.079e-06 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124188 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644969+0000", 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016553924000000001, 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018085014, 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124188 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644969+0000", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644969+0000", 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644969+0000", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644971+0000", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1299999999999999e-06 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644978+0000", 2024-04-23T21:22:23.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1959999999999996e-06 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644983+0000", 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9640000000000002e-06 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124413 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649135+0000", 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012388045, 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013974515, 2024-04-23T21:22:23.783 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124413 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649135+0000", 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649135+0000", 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649135+0000", 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0100000000000005e-07 2024-04-23T21:22:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649139+0000", 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4960000000000001e-06 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649146+0000", 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2840000000000001e-06 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649152+0000", 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.711e-06 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124475 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651639+0000", 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0098833080000000004, 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011524596, 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.787 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124475 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651639+0000", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651639+0000", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651640+0000", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6600000000000002e-07 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651642+0000", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9769999999999999e-06 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651647+0000", 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3519999999999997e-06 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651652+0000", 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8949999999999997e-06 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124539 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.790 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655043+0000", 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0064800609999999996, 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081754480000000001, 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124539 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655043+0000", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655043+0000", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655043+0000", 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8599999999999998e-07 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.792 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655046+0000", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5229999999999998e-06 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655052+0000", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0589999999999999e-06 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655057+0000", 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5679999999999999e-06 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123960 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643232+0000", 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018290484999999999, 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020025396000000001, 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123960 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.794 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643232+0000", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643232+0000", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643233+0000", 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:23.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643235+0000", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4509999999999999e-06 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643243+0000", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3849999999999996e-06 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643248+0000", 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.609e-06 2024-04-23T21:22:23.796 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124077 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644184+0000", 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017338499, 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019101104000000001, 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.797 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124077 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644184+0000", 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644184+0000", 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644185+0000", 2024-04-23T21:22:23.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0200000000000003e-07 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644187+0000", 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644194+0000", 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9659999999999999e-06 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644199+0000", 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.614e-06 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124189 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644989+0000", 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016533283999999999, 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018352771, 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.801 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124189 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644989+0000", 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644989+0000", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644990+0000", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644992+0000", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2199999999999999e-06 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645001+0000", 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8880000000000003e-06 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645006+0000", 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8380000000000001e-06 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.804 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124288 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645872+0000", 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015651004, 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01749769, 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.805 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124288 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645872+0000", 2024-04-23T21:22:23.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645872+0000", 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645872+0000", 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5400000000000002e-07 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645894+0000", 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2314999999999999e-05 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645902+0000", 2024-04-23T21:22:23.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0360000000000006e-06 2024-04-23T21:22:23.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645909+0000", 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5610000000000004e-06 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124476 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651661+0000", 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0098616139999999994, 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011736327, 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.812 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.813 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124476 2024-04-23T21:22:23.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651661+0000", 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651661+0000", 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651662+0000", 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2300000000000002e-07 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651664+0000", 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1670000000000002e-06 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651672+0000", 2024-04-23T21:22:23.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8140000000000001e-06 2024-04-23T21:22:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651676+0000", 2024-04-23T21:22:23.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7160000000000002e-06 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.819 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124541 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655064+0000", 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0064590359999999996, 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0083613669999999998, 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.820 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124541 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.821 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655064+0000", 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655064+0000", 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655064+0000", 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1e-07 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655076+0000", 2024-04-23T21:22:23.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2194999999999999e-05 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655082+0000", 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2280000000000003e-06 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655087+0000", 2024-04-23T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8500000000000002e-06 2024-04-23T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.826 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123975 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643256+0000", 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018266513000000002, 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020204913000000001, 2024-04-23T21:22:23.827 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123975 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643256+0000", 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643256+0000", 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643257+0000", 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0999999999999998e-07 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643259+0000", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2469999999999998e-06 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643265+0000", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.297e-06 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643270+0000", 2024-04-23T21:22:23.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0000000000000004e-06 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124081 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644206+0000", 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017316802999999999, 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019308209, 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124081 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644206+0000", 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644206+0000", 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644206+0000", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4000000000000002e-07 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644214+0000", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1040000000000001e-06 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644220+0000", 2024-04-23T21:22:23.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8199999999999999e-06 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644225+0000", 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9219999999999996e-06 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124202 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645032+0000", 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016491038, 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018540340999999998, 2024-04-23T21:22:23.835 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124202 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645032+0000", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645032+0000", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645032+0000", 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.89e-07 2024-04-23T21:22:23.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645035+0000", 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.537e-06 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645043+0000", 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3410000000000003e-06 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645049+0000", 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7309999999999998e-06 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124289 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645918+0000", 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015605028, 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017709140000000002, 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.841 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124289 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645918+0000", 2024-04-23T21:22:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645918+0000", 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645918+0000", 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7999999999999996e-07 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645920+0000", 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.12e-06 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645932+0000", 2024-04-23T21:22:23.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2104e-05 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645938+0000", 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7579999999999998e-06 2024-04-23T21:22:23.847 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124477 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651682+0000", 2024-04-23T21:22:23.848 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.009840494, 2024-04-23T21:22:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012000237, 2024-04-23T21:22:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124477 2024-04-23T21:22:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.850 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651682+0000", 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651682+0000", 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651683+0000", 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651687+0000", 2024-04-23T21:22:23.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.605e-06 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651693+0000", 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6339999999999998e-06 2024-04-23T21:22:23.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651698+0000", 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9590000000000003e-06 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.855 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124543 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655120+0000", 2024-04-23T21:22:23.856 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0064026040000000001, 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0086200310000000002, 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124543 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655120+0000", 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655120+0000", 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655121+0000", 2024-04-23T21:22:23.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3799999999999997e-07 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655123+0000", 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7530000000000002e-06 2024-04-23T21:22:23.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655130+0000", 2024-04-23T21:22:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7220000000000002e-06 2024-04-23T21:22:23.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655136+0000", 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7189999999999998e-06 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.863 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124735 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661225+0000", 2024-04-23T21:22:23.864 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00029790799999999999, 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025693339999999999, 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.865 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124735 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661225+0000", 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661225+0000", 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661225+0000", 2024-04-23T21:22:23.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661227+0000", 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661233+0000", 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5130000000000004e-06 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661238+0000", 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6460000000000001e-06 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123976 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643277+0000", 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018245838, 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020575547, 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.871 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123976 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643277+0000", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643277+0000", 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643278+0000", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7999999999999995e-07 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643280+0000", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2000000000000001e-06 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643286+0000", 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3980000000000004e-06 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643291+0000", 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1599999999999997e-06 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124082 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644232+0000", 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017290967000000001, 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019674703000000002, 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124082 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644232+0000", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644232+0000", 2024-04-23T21:22:23.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644232+0000", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644234+0000", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.131e-06 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644242+0000", 2024-04-23T21:22:23.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6769999999999995e-06 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644247+0000", 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.434e-06 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124203 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645058+0000", 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016465275000000001, 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018910856, 2024-04-23T21:22:23.877 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124203 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645058+0000", 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645058+0000", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645058+0000", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6699999999999999e-07 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645060+0000", 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3989999999999998e-06 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645067+0000", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.669e-06 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645073+0000", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999997e-06 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124351 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648022+0000", 2024-04-23T21:22:23.880 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013500318000000001, 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016001971, 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124351 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648022+0000", 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648022+0000", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648023+0000", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648025+0000", 2024-04-23T21:22:23.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9410000000000001e-06 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648031+0000", 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7540000000000003e-06 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648035+0000", 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6090000000000003e-06 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124419 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649374+0000", 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012148459, 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014708712000000001, 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124419 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649374+0000", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649374+0000", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649375+0000", 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4000000000000002e-07 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649377+0000", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649385+0000", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8310000000000001e-06 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649390+0000", 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8729999999999998e-06 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.886 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124737 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661244+0000", 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.000279268, 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0028921210000000001, 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.887 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124737 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661244+0000", 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661244+0000", 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661244+0000", 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661246+0000", 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1500000000000002e-06 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661252+0000", 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5860000000000004e-06 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661256+0000", 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6199999999999998e-06 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123977 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643304+0000", 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018219012, 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020889462000000001, 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.890 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123977 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643304+0000", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643304+0000", 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643304+0000", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4899999999999995e-07 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643307+0000", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3099999999999999e-06 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643313+0000", 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6619999999999999e-06 2024-04-23T21:22:23.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643319+0000", 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6779999999999997e-06 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124088 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644254+0000", 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017268467999999999, 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019993449, 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.893 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124088 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644254+0000", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644254+0000", 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644255+0000", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644257+0000", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644264+0000", 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6409999999999996e-06 2024-04-23T21:22:23.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644269+0000", 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1590000000000001e-06 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124209 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645080+0000", 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016442783999999998, 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019225335, 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.896 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124209 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645080+0000", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645080+0000", 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645080+0000", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8500000000000002e-07 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645083+0000", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1789999999999998e-06 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645090+0000", 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3050000000000004e-06 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645096+0000", 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5939999999999999e-06 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124352 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648041+0000", 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013481503000000001, 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016317284000000001, 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124352 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648041+0000", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648041+0000", 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648042+0000", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648043+0000", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7889999999999999e-06 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648049+0000", 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6380000000000001e-06 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648054+0000", 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9110000000000001e-06 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124420 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649396+0000", 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012127043000000001, 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015020383, 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.902 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124420 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649396+0000", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649396+0000", 2024-04-23T21:22:23.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649396+0000", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649398+0000", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.012e-06 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649404+0000", 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5720000000000002e-06 2024-04-23T21:22:23.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649409+0000", 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3510000000000001e-06 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124678 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659536+0000", 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0019869240000000002, 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0049344619999999997, 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.905 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124678 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659536+0000", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659536+0000", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659536+0000", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2099999999999998e-07 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659538+0000", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.136e-06 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659544+0000", 2024-04-23T21:22:23.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.784e-06 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659549+0000", 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0379999999999999e-06 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124739 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661262+0000", 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00026119400000000002, 2024-04-23T21:22:23.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032671589999999999, 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124739 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661262+0000", 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661262+0000", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661262+0000", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661264+0000", 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1799999999999999e-06 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661270+0000", 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8769999999999998e-06 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661664+0000", 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000393543 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123978 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.911 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643337+0000", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018185509999999998, 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021248091, 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123978 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643337+0000", 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643337+0000", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643338+0000", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7800000000000001e-07 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643340+0000", 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2390000000000001e-06 2024-04-23T21:22:23.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643349+0000", 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1849999999999993e-06 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643355+0000", 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5450000000000003e-06 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.914 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124090 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644297+0000", 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017225467000000001, 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020343963, 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124090 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644297+0000", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644297+0000", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644298+0000", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2200000000000004e-07 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644301+0000", 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6649999999999999e-06 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644308+0000", 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6120000000000001e-06 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644314+0000", 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5389999999999998e-06 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124210 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645103+0000", 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016419349999999999, 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019597348000000001, 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124210 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645103+0000", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645103+0000", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645104+0000", 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.15e-07 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645106+0000", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645113+0000", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5490000000000003e-06 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645118+0000", 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0520000000000001e-06 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.920 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124295 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646448+0000", 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015074825, 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018307054999999999, 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.921 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124295 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646448+0000", 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646448+0000", 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646449+0000", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646452+0000", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3869999999999999e-06 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646460+0000", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1189999999999997e-06 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646466+0000", 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1510000000000001e-06 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124353 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648060+0000", 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01346295, 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016753146999999999, 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124353 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648060+0000", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648060+0000", 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648060+0000", 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.91e-07 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648062+0000", 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8610000000000001e-06 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648067+0000", 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3809999999999998e-06 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648072+0000", 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7879999999999997e-06 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124484 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651780+0000", 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0097431050000000002, 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013088700999999999, 2024-04-23T21:22:23.927 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124484 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651780+0000", 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651780+0000", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651780+0000", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651782+0000", 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651788+0000", 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.947e-06 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652234+0000", 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044588100000000002 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124621 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.930 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.657136+0000", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0043869399999999998, 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0077955050000000003, 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124621 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657136+0000", 2024-04-23T21:22:23.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657136+0000", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657136+0000", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.08e-07 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657146+0000", 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0173000000000001e-05 2024-04-23T21:22:23.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657152+0000", 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8109999999999999e-06 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657157+0000", 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124680 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.933 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659555+0000", 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0019679379999999998, 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.005431661, 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124680 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659555+0000", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659555+0000", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659555+0000", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659557+0000", 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659563+0000", 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2580000000000003e-06 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659567+0000", 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8609999999999997e-06 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124742 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661674+0000", 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9998484, 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0033623120000000001, 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124742 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661674+0000", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661674+0000", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661675+0000", 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661678+0000", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2310000000000001e-06 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661685+0000", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0199999999999997e-06 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661690+0000", 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8269999999999997e-06 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.939 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123979 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643371+0000", 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018151423, 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021694260999999999, 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123979 2024-04-23T21:22:23.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643371+0000", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643371+0000", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643372+0000", 2024-04-23T21:22:23.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6300000000000005e-07 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643374+0000", 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3489999999999999e-06 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643382+0000", 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8420000000000005e-06 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643388+0000", 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1290000000000001e-06 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124102 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644321+0000", 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017201655, 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020824371000000001, 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124102 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644321+0000", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644321+0000", 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644322+0000", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4799999999999999e-07 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644324+0000", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1950000000000002e-06 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644330+0000", 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4010000000000002e-06 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644335+0000", 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0640000000000001e-06 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124215 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645125+0000", 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016397694000000001, 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020068459, 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.946 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124215 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645125+0000", 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645125+0000", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645125+0000", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645135+0000", 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3719999999999994e-06 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645148+0000", 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3395e-05 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645154+0000", 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9819999999999996e-06 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124296 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.949 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646477+0000", 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015045820999999999, 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018773100000000001, 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124296 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646477+0000", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646477+0000", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646478+0000", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646480+0000", 2024-04-23T21:22:23.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3870000000000002e-06 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646487+0000", 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9099999999999999e-06 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646493+0000", 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124362 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648078+0000", 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013444793, 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017227247000000001, 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124362 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648078+0000", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648078+0000", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648078+0000", 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.58e-07 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648080+0000", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8959999999999999e-06 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648086+0000", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6210000000000001e-06 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648091+0000", 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8620000000000002e-06 2024-04-23T21:22:23.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124485 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652551+0000", 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0089716369999999993, 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012810828, 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124485 2024-04-23T21:22:23.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652551+0000", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652551+0000", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652552+0000", 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2699999999999999e-07 2024-04-23T21:22:23.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652556+0000", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7910000000000002e-06 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652564+0000", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1480000000000006e-06 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652571+0000", 2024-04-23T21:22:23.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9539999999999998e-06 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124623 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.657164+0000", 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0043587590000000002, 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082477990000000001, 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124623 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657164+0000", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657164+0000", 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657164+0000", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657166+0000", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9649999999999998e-06 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657172+0000", 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5559999999999998e-06 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657177+0000", 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5940000000000004e-06 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123980 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643398+0000", 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018124537999999999, 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022049831999999998, 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.962 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123980 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643398+0000", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643398+0000", 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643399+0000", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.39e-07 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643401+0000", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4159999999999998e-06 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643408+0000", 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6869999999999997e-06 2024-04-23T21:22:23.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643414+0000", 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8730000000000004e-06 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124103 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644342+0000", 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017181029, 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021133534999999998, 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.965 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124103 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644342+0000", 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644342+0000", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644342+0000", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.39e-07 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644344+0000", 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1019999999999999e-06 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644352+0000", 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5719999999999997e-06 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644357+0000", 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0259999999999998e-06 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124216 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645163+0000", 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016359756999999999, 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020338723, 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124216 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645163+0000", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645163+0000", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645164+0000", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645173+0000", 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4520000000000003e-06 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645186+0000", 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3230999999999999e-05 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645193+0000", 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124363 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648096+0000", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013426544, 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01744306, 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124363 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648096+0000", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648096+0000", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648097+0000", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.77e-07 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648099+0000", 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.911e-06 2024-04-23T21:22:23.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648104+0000", 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648109+0000", 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5360000000000003e-06 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.974 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124425 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.977 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649465+0000", 2024-04-23T21:22:23.977 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012058049, 2024-04-23T21:22:23.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016128877999999999, 2024-04-23T21:22:23.977 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124425 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649465+0000", 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649465+0000", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649465+0000", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2500000000000001e-07 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649467+0000", 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.844e-06 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649473+0000", 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9379999999999997e-06 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649477+0000", 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124624 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.657182+0000", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0043407100000000002, 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0084685679999999992, 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124624 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657182+0000", 2024-04-23T21:22:23.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657182+0000", 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657961+0000", 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000778562 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657965+0000", 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657978+0000", 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140999999999999e-05 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.657985+0000", 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7000000000000008e-06 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123990 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643421+0000", 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018101973, 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022286653999999999, 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123990 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643421+0000", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643421+0000", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643421+0000", 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1300000000000001e-07 2024-04-23T21:22:23.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643423+0000", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.091e-06 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643429+0000", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1199999999999999e-06 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643434+0000", 2024-04-23T21:22:23.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7439999999999998e-06 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124104 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644367+0000", 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017155976999999999, 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021397860000000001, 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.987 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124104 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644367+0000", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644367+0000", 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644367+0000", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644369+0000", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644377+0000", 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2899999999999997e-06 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644382+0000", 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9849999999999997e-06 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124219 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645210+0000", 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016312665, 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020608663999999999, 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.990 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124219 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645210+0000", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645210+0000", 2024-04-23T21:22:23.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645211+0000", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2699999999999999e-07 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645213+0000", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8420000000000001e-06 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645221+0000", 2024-04-23T21:22:23.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2529999999999998e-06 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645233+0000", 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1855000000000001e-05 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124426 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649484+0000", 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012039245000000001, 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016393385, 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.993 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124426 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649484+0000", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649484+0000", 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649484+0000", 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0400000000000002e-07 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649486+0000", 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8700000000000001e-06 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649492+0000", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7950000000000004e-06 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649496+0000", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6550000000000003e-06 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124556 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655246+0000", 2024-04-23T21:22:23.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0062768260000000001, 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010689847000000001, 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124556 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655246+0000", 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655246+0000", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655246+0000", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655248+0000", 2024-04-23T21:22:23.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9980000000000002e-06 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655255+0000", 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749999999999998e-06 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655260+0000", 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:23.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124683 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659616+0000", 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0019069359999999999, 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0063769120000000002, 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124683 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659616+0000", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659616+0000", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659616+0000", 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3999999999999997e-07 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659618+0000", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0679999999999999e-06 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659624+0000", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0599999999999996e-06 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659629+0000", 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.78e-06 2024-04-23T21:22:24.002 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123991 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643441+0000", 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018082033000000001, 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022606753, 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123991 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643441+0000", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643441+0000", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643441+0000", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.27e-07 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643443+0000", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643451+0000", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4970000000000004e-06 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643456+0000", 2024-04-23T21:22:24.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0799999999999996e-06 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124109 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644388+0000", 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017134712, 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021716626999999999, 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124109 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644388+0000", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644388+0000", 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644388+0000", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.65e-07 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644390+0000", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.057e-06 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644398+0000", 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5510000000000002e-06 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644409+0000", 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0532e-05 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124220 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645274+0000", 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016249258999999999, 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020885593000000001, 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.009 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124220 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645274+0000", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645274+0000", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645274+0000", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645277+0000", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8049999999999998e-06 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645285+0000", 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2449999999999998e-06 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645294+0000", 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7849999999999997e-06 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124301 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646542+0000", 2024-04-23T21:22:24.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014980922000000001, 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019675378, 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124301 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646542+0000", 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646542+0000", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646542+0000", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646544+0000", 2024-04-23T21:22:24.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0899999999999999e-06 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646551+0000", 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3470000000000004e-06 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646556+0000", 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.853e-06 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124497 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652635+0000", 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0088875729999999993, 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013639517, 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124497 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652635+0000", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652635+0000", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652636+0000", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9299999999999998e-07 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652643+0000", 2024-04-23T21:22:24.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8990000000000004e-06 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652648+0000", 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8939999999999998e-06 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652653+0000", 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7859999999999996e-06 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124557 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655266+0000", 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0062572699999999997, 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011066822, 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124557 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655266+0000", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655266+0000", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655266+0000", 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.44e-07 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655268+0000", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.029e-06 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655274+0000", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655278+0000", 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4519999999999999e-06 2024-04-23T21:22:24.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124626 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658030+0000", 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0034925949999999998, 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0083559159999999997, 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.022 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124626 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658030+0000", 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658030+0000", 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658031+0000", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658033+0000", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6749999999999998e-06 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658040+0000", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7970000000000003e-06 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658046+0000", 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.19e-06 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124686 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659635+0000", 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0018877830000000001, 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.006795054, 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124686 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659635+0000", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659635+0000", 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659635+0000", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.75e-07 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659645+0000", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0929999999999992e-06 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659651+0000", 2024-04-23T21:22:24.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9399999999999999e-06 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659656+0000", 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0669999999999999e-06 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:123994 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643462+0000", 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018060772999999999, 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023003399000000001, 2024-04-23T21:22:24.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 123994 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643462+0000", 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643462+0000", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643462+0000", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.58e-07 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643472+0000", 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4679999999999998e-06 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643479+0000", 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8850000000000001e-06 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643484+0000", 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3179999999999997e-06 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.031 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124110 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644417+0000", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017105482000000002, 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022074824, 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124110 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644417+0000", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644417+0000", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644418+0000", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644420+0000", 2024-04-23T21:22:24.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5940000000000001e-06 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644427+0000", 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7880000000000001e-06 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644432+0000", 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3199999999999999e-06 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124221 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645303+0000", 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016219859, 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021217406000000001, 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124221 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645303+0000", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645303+0000", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645303+0000", 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1100000000000001e-07 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645306+0000", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4619999999999999e-06 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645312+0000", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6930000000000001e-06 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645318+0000", 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3190000000000002e-06 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124302 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646571+0000", 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014951409000000001, 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019975639999999999, 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124302 2024-04-23T21:22:24.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646571+0000", 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646571+0000", 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646572+0000", 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646578+0000", 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5450000000000003e-06 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646586+0000", 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3310000000000004e-06 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646591+0000", 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5300000000000004e-06 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124368 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648484+0000", 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013038451, 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018090605999999999, 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124368 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648484+0000", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648484+0000", 2024-04-23T21:22:24.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648485+0000", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4999999999999998e-07 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648487+0000", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0559999999999999e-06 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648493+0000", 2024-04-23T21:22:24.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0789999999999997e-06 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648498+0000", 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8620000000000002e-06 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124498 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652660+0000", 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0088631379999999996, 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013941853000000001, 2024-04-23T21:22:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124498 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652660+0000", 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652660+0000", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652660+0000", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652662+0000", 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0899999999999999e-06 2024-04-23T21:22:24.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652667+0000", 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0159999999999999e-06 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652672+0000", 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7829999999999998e-06 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.047 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124558 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655284+0000", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0062383359999999997, 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011344709999999999, 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124558 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655284+0000", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655284+0000", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655285+0000", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999999999999997e-07 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655287+0000", 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0669999999999999e-06 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655293+0000", 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.139e-06 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655298+0000", 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1309999999999997e-06 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124001 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643491+0000", 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018031314999999999, 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023184742000000001, 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124001 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643491+0000", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643491+0000", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643492+0000", 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:24.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643494+0000", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.238e-06 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643500+0000", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0510000000000002e-06 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643515+0000", 2024-04-23T21:22:24.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4676e-05 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124116 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644440+0000", 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017083009999999999, 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022293054, 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124116 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644440+0000", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644440+0000", 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644440+0000", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5799999999999999e-07 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644442+0000", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1430000000000001e-06 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644449+0000", 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9600000000000003e-06 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644455+0000", 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.096e-06 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124229 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645325+0000", 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016197639, 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021462371000000001, 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124229 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645325+0000", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645325+0000", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645326+0000", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9399999999999995e-07 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645328+0000", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645334+0000", 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.173e-06 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645339+0000", 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9840000000000001e-06 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124310 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646600+0000", 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014923267, 2024-04-23T21:22:24.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020246151, 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124310 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646600+0000", 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646600+0000", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646600+0000", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646614+0000", 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3925e-05 2024-04-23T21:22:24.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646621+0000", 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2949999999999996e-06 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646630+0000", 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5499999999999995e-06 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.063 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124369 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648504+0000", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013018775, 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018385747000000001, 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124369 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648504+0000", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648504+0000", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648504+0000", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3799999999999998e-07 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648506+0000", 2024-04-23T21:22:24.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.883e-06 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648512+0000", 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7910000000000001e-06 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648517+0000", 2024-04-23T21:22:24.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6970000000000001e-06 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124559 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655304+0000", 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0062188, 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011616151999999999, 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124559 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655304+0000", 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655304+0000", 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655304+0000", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.03e-07 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655306+0000", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655312+0000", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8830000000000003e-06 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655317+0000", 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.079e-06 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124752 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661804+0000", 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9997187, 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0051511550000000001, 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124752 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661804+0000", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661804+0000", 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661804+0000", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661806+0000", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661812+0000", 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0249999999999999e-06 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661817+0000", 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6750000000000001e-06 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124002 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643522+0000", 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.018000324000000002, 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023465906000000002, 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.073 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124002 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643522+0000", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643522+0000", 2024-04-23T21:22:24.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643523+0000", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1200000000000003e-07 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643525+0000", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3690000000000001e-06 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643532+0000", 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8249999999999999e-06 2024-04-23T21:22:24.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643539+0000", 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.764e-06 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124117 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644461+0000", 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017061748000000002, 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022567523999999999, 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.076 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124117 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644461+0000", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644461+0000", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644461+0000", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7399999999999999e-07 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644464+0000", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644470+0000", 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3330000000000002e-06 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644475+0000", 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8980000000000004e-06 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124230 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.079 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645349+0000", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01617391, 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021736405, 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124230 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645349+0000", 2024-04-23T21:22:24.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645349+0000", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645349+0000", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645352+0000", 2024-04-23T21:22:24.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.52e-06 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645358+0000", 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5989999999999998e-06 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645364+0000", 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4330000000000003e-06 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124311 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646637+0000", 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014885772, 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020502421999999999, 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124311 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646637+0000", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646637+0000", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646637+0000", 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5400000000000002e-07 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646642+0000", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2470000000000002e-06 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646652+0000", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0051000000000001e-05 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646659+0000", 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3309999999999998e-06 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.085 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124370 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648523+0000", 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013000047000000001, 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018669584999999999, 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124370 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.086 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648523+0000", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648523+0000", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648523+0000", 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.53e-07 2024-04-23T21:22:24.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648525+0000", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.916e-06 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648532+0000", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7399999999999998e-06 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648536+0000", 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4630000000000003e-06 2024-04-23T21:22:24.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124438 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649574+0000", 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011949111, 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017671235, 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.089 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124438 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649574+0000", 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649574+0000", 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649574+0000", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3999999999999997e-07 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649576+0000", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9039999999999999e-06 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649582+0000", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649586+0000", 2024-04-23T21:22:24.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6859999999999997e-06 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124562 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655323+0000", 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0061993819999999998, 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011949329999999999, 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.092 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124562 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655323+0000", 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655323+0000", 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655324+0000", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5100000000000001e-07 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655326+0000", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0329999999999998e-06 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655332+0000", 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3010000000000003e-06 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655337+0000", 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7670000000000003e-06 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124691 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659701+0000", 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0018220440000000001, 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0076000590000000002, 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.095 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124691 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659701+0000", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659701+0000", 2024-04-23T21:22:24.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659701+0000", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659703+0000", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0959999999999999e-06 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659709+0000", 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:24.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659713+0000", 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5750000000000002e-06 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124754 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661823+0000", 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9996996, 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0055044949999999999, 2024-04-23T21:22:24.098 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124754 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661823+0000", 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661823+0000", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661824+0000", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661830+0000", 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4420000000000003e-06 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661836+0000", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8170000000000004e-06 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661841+0000", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7949999999999998e-06 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124004 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643546+0000", 2024-04-23T21:22:24.101 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017977125, 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023811876999999999, 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124004 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643546+0000", 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643546+0000", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643546+0000", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0200000000000002e-07 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643548+0000", 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643555+0000", 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643560+0000", 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.575e-06 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124118 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.104 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644482+0000", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017040362, 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022924388, 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124118 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644482+0000", 2024-04-23T21:22:24.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644482+0000", 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644483+0000", 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644485+0000", 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1940000000000001e-06 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644491+0000", 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.409e-06 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644497+0000", 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0919999999999997e-06 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124312 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646665+0000", 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014857521, 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020798463, 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124312 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.108 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646665+0000", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646665+0000", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646666+0000", 2024-04-23T21:22:24.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4299999999999999e-07 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646668+0000", 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.063e-06 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646674+0000", 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0410000000000003e-06 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646679+0000", 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.831e-06 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124440 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649592+0000", 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011930924000000001, 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017926223000000002, 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.111 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124440 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649592+0000", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649592+0000", 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649592+0000", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4799999999999999e-07 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649594+0000", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9230000000000001e-06 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649600+0000", 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5480000000000001e-06 2024-04-23T21:22:24.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649604+0000", 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5329999999999996e-06 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124503 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652722+0000", 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0088011119999999998, 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014852712000000001, 2024-04-23T21:22:24.114 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124503 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652722+0000", 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652722+0000", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652722+0000", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652724+0000", 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3149999999999999e-06 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652730+0000", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9739999999999999e-06 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652736+0000", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9730000000000002e-06 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124692 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659719+0000", 2024-04-23T21:22:24.117 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0018042500000000001, 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0079094300000000003, 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124692 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659719+0000", 2024-04-23T21:22:24.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659719+0000", 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659719+0000", 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4900000000000001e-07 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659721+0000", 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9319999999999999e-06 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659727+0000", 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6030000000000004e-06 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659732+0000", 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.181e-06 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.120 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124761 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661847+0000", 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9996758, 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00583908, 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124761 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661847+0000", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661847+0000", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661848+0000", 2024-04-23T21:22:24.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1900000000000002e-07 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661850+0000", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1749999999999999e-06 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661855+0000", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5450000000000003e-06 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661860+0000", 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5759999999999999e-06 2024-04-23T21:22:24.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124005 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643567+0000", 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017955597, 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024175235, 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124005 2024-04-23T21:22:24.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643567+0000", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643567+0000", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643568+0000", 2024-04-23T21:22:24.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3200000000000005e-07 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643570+0000", 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5129999999999999e-06 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643578+0000", 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9440000000000005e-06 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643583+0000", 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2360000000000004e-06 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124125 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644503+0000", 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017019958000000002, 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023295680999999999, 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124125 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644503+0000", 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644503+0000", 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644503+0000", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644510+0000", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644516+0000", 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3509999999999998e-06 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644522+0000", 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1290000000000001e-06 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124237 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645401+0000", 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016121361000000001, 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022440857000000002, 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.130 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124237 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645401+0000", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645401+0000", 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645402+0000", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9500000000000003e-07 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645404+0000", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.508e-06 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645411+0000", 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7179999999999999e-06 2024-04-23T21:22:24.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645416+0000", 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2220000000000001e-06 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124504 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652742+0000", 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0087805829999999998, 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01513197, 2024-04-23T21:22:24.133 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124504 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652742+0000", 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652742+0000", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652743+0000", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4099999999999999e-07 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652745+0000", 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9939999999999999e-06 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652751+0000", 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4359999999999999e-06 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652756+0000", 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.707e-06 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124572 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.136 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655389+0000", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0061338399999999998, 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012512095000000001, 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124572 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655389+0000", 2024-04-23T21:22:24.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655389+0000", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655389+0000", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7999999999999996e-07 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655392+0000", 2024-04-23T21:22:24.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4279999999999999e-06 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655399+0000", 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.781e-06 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655404+0000", 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9200000000000003e-06 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124636 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658164+0000", 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033586979999999998, 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0097689620000000008, 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124636 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658164+0000", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658164+0000", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658164+0000", 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658167+0000", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658172+0000", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.677e-06 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658177+0000", 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5469999999999998e-06 2024-04-23T21:22:24.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124694 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659738+0000", 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017844880000000001, 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.008222877, 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124694 2024-04-23T21:22:24.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659738+0000", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659738+0000", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659739+0000", 2024-04-23T21:22:24.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8799999999999998e-07 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659741+0000", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.418e-06 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659747+0000", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.631e-06 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659751+0000", 2024-04-23T21:22:24.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4399999999999998e-06 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124763 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661865+0000", 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9996576, 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0061418719999999996, 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124763 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661865+0000", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661865+0000", 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661866+0000", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2499999999999995e-07 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661868+0000", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661874+0000", 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7409999999999998e-06 2024-04-23T21:22:24.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661878+0000", 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6380000000000003e-06 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124007 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643597+0000", 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017925832999999999, 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024467610000000001, 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.149 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124007 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643597+0000", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643597+0000", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643597+0000", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2099999999999997e-07 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643600+0000", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4159999999999998e-06 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643607+0000", 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7689999999999999e-06 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643612+0000", 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3399999999999997e-06 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124126 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644529+0000", 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016994003000000001, 2024-04-23T21:22:24.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023589822999999999, 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124126 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644529+0000", 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644529+0000", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644529+0000", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644531+0000", 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2170000000000001e-06 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644538+0000", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9739999999999996e-06 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644543+0000", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8420000000000004e-06 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124238 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645424+0000", 2024-04-23T21:22:24.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016099104999999999, 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022751460000000001, 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124238 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645424+0000", 2024-04-23T21:22:24.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645424+0000", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645424+0000", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2800000000000002e-07 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645426+0000", 2024-04-23T21:22:24.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1950000000000002e-06 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645432+0000", 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1820000000000003e-06 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645442+0000", 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1500000000000005e-06 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124505 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652761+0000", 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0087614019999999997, 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015465840999999999, 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124505 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652761+0000", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652761+0000", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652762+0000", 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652764+0000", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.058e-06 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652770+0000", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7570000000000001e-06 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652774+0000", 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7160000000000002e-06 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124574 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655410+0000", 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0061131670000000001, 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012963203, 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.162 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124574 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655410+0000", 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655410+0000", 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655410+0000", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2500000000000001e-07 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655412+0000", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2419999999999999e-06 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655419+0000", 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4389999999999997e-06 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655424+0000", 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.011e-06 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124637 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658183+0000", 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033402190000000002, 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010218043, 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124637 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658183+0000", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658183+0000", 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658183+0000", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1800000000000002e-07 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658185+0000", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9120000000000001e-06 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658190+0000", 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5110000000000003e-06 2024-04-23T21:22:24.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658195+0000", 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8670000000000002e-06 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124696 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659757+0000", 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017656410000000001, 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0086851930000000008, 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124696 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659757+0000", 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659757+0000", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659758+0000", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8500000000000001e-07 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659760+0000", 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9010000000000001e-06 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659766+0000", 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4139999999999999e-06 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659771+0000", 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124009 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.171 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643631+0000", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017892254, 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024880962999999999, 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124009 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643631+0000", 2024-04-23T21:22:24.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643631+0000", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643631+0000", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0800000000000004e-07 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643634+0000", 2024-04-23T21:22:24.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6529999999999998e-06 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643641+0000", 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1380000000000001e-06 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643646+0000", 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3159999999999996e-06 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124130 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644549+0000", 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016973511, 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024020651000000001, 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124130 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644549+0000", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644549+0000", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644550+0000", 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0200000000000003e-07 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644552+0000", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1639999999999999e-06 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644558+0000", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.19e-06 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644563+0000", 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7709999999999997e-06 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.177 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124317 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646772+0000", 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014750350000000001, 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021851463000000002, 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124317 2024-04-23T21:22:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.181 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646772+0000", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646772+0000", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646773+0000", 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7599999999999997e-07 2024-04-23T21:22:24.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646776+0000", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.819e-06 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646783+0000", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0890000000000002e-06 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646789+0000", 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2609999999999999e-06 2024-04-23T21:22:24.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124376 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648607+0000", 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012915754999999999, 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020073885, 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.184 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124376 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648607+0000", 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648607+0000", 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648607+0000", 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648610+0000", 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2479999999999999e-06 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648615+0000", 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5380000000000002e-06 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648621+0000", 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.981e-06 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124445 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649650+0000", 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011873154, 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019094363, 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124445 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649650+0000", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649650+0000", 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649650+0000", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.9200000000000002e-07 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649652+0000", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.108e-06 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649658+0000", 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8950000000000003e-06 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649662+0000", 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4739999999999999e-06 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124639 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658201+0000", 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033220540000000001, 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010583421000000001, 2024-04-23T21:22:24.190 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124639 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658201+0000", 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658201+0000", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658201+0000", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2500000000000001e-07 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658203+0000", 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658210+0000", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3820000000000001e-06 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658214+0000", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7539999999999997e-06 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124697 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659776+0000", 2024-04-23T21:22:24.193 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0017464329999999999, 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0090346000000000003, 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124697 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659776+0000", 2024-04-23T21:22:24.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659776+0000", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659777+0000", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659782+0000", 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4009999999999996e-06 2024-04-23T21:22:24.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659788+0000", 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.947e-06 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659793+0000", 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6419999999999998e-06 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124024 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643653+0000", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017869548999999998, 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025191563, 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124024 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643653+0000", 2024-04-23T21:22:24.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643653+0000", 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643654+0000", 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4300000000000003e-07 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643656+0000", 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1229999999999998e-06 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643662+0000", 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1099999999999999e-06 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643667+0000", 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9239999999999998e-06 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124131 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644569+0000", 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016953614999999998, 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024330516999999999, 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124131 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644569+0000", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644569+0000", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644570+0000", 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3799999999999997e-07 2024-04-23T21:22:24.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644572+0000", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0200000000000001e-06 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644583+0000", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.133e-05 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644588+0000", 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:24.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124318 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.646797+0000", 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014726134, 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022169991, 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.203 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124318 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646797+0000", 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646797+0000", 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646797+0000", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4999999999999998e-07 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646799+0000", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1579999999999999e-06 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646806+0000", 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7340000000000002e-06 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.646812+0000", 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0569999999999998e-06 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124377 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648629+0000", 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012894184, 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020396436, 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.206 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124377 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648629+0000", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648629+0000", 2024-04-23T21:22:24.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648629+0000", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.65e-07 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648631+0000", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648637+0000", 2024-04-23T21:22:24.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8109999999999999e-06 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648641+0000", 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5839999999999996e-06 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124446 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649668+0000", 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011854599, 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019409874000000001, 2024-04-23T21:22:24.209 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124446 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649668+0000", 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649668+0000", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649669+0000", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9299999999999999e-07 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649679+0000", 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0018e-05 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649685+0000", 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2929999999999997e-06 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649690+0000", 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3380000000000004e-06 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124640 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.212 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658221+0000", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0033020879999999999, 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.01091345, 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124640 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658221+0000", 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658221+0000", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658221+0000", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6600000000000002e-07 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658227+0000", 2024-04-23T21:22:24.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2700000000000001e-06 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658233+0000", 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7690000000000002e-06 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658241+0000", 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3080000000000008e-06 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124699 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659799+0000", 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00172349, 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0093635739999999995, 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124699 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659799+0000", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659799+0000", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659800+0000", 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8299999999999997e-07 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659802+0000", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.956e-06 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659807+0000", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5589999999999996e-06 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659812+0000", 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6330000000000004e-06 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.218 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124773 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663395+0000", 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9981275, 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0057969639999999999, 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124773 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.219 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663395+0000", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663395+0000", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663396+0000", 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:24.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663399+0000", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1049999999999999e-06 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663406+0000", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1890000000000001e-06 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663412+0000", 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8220000000000003e-06 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.221 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124026 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643701+0000", 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017821335000000001, 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025529796, 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.222 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124026 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643701+0000", 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643701+0000", 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643703+0000", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.699e-06 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643706+0000", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5979999999999999e-06 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643713+0000", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643719+0000", 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5940000000000001e-06 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124132 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644595+0000", 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016927641, 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024679366000000001, 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124132 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644595+0000", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644595+0000", 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644596+0000", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644598+0000", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.221e-06 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644604+0000", 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0920000000000003e-06 2024-04-23T21:22:24.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644608+0000", 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6399999999999996e-06 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124447 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649698+0000", 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011824886999999999, 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.019609167, 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.228 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124447 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649698+0000", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649698+0000", 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649698+0000", 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649700+0000", 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649706+0000", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6729999999999997e-06 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649710+0000", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.464e-06 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124641 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658248+0000", 2024-04-23T21:22:24.231 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0032745579999999999, 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011124366, 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124641 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658248+0000", 2024-04-23T21:22:24.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658248+0000", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658249+0000", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658251+0000", 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0030000000000001e-06 2024-04-23T21:22:24.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658343+0000", 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2347000000000002e-05 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658353+0000", 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0001e-05 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124708 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659817+0000", 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001705367, 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0096094300000000004, 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124708 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659817+0000", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659817+0000", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659818+0000", 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5400000000000002e-07 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659820+0000", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659825+0000", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4500000000000003e-06 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659830+0000", 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5290000000000002e-06 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.237 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124774 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663432+0000", 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9980903, 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0060513659999999999, 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124774 2024-04-23T21:22:24.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663432+0000", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663432+0000", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663433+0000", 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7599999999999997e-07 2024-04-23T21:22:24.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663435+0000", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.463e-06 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663442+0000", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.985e-06 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663448+0000", 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:24.240 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124028 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643727+0000", 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01779613, 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025812743999999999, 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.241 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124028 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643727+0000", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643727+0000", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643727+0000", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5400000000000001e-07 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643730+0000", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3350000000000001e-06 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643736+0000", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4400000000000002e-06 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643741+0000", 2024-04-23T21:22:24.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1290000000000004e-06 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124137 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644615+0000", 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016907643, 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024984129000000001, 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.244 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124137 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644615+0000", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644615+0000", 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644616+0000", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6800000000000001e-07 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644618+0000", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.154e-06 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644624+0000", 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.297e-06 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644641+0000", 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6943e-05 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124246 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645537+0000", 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015985927, 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024090529999999999, 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.247 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124246 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645537+0000", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645537+0000", 2024-04-23T21:22:24.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645537+0000", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3799999999999998e-07 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645540+0000", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3860000000000001e-06 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645546+0000", 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7059999999999998e-06 2024-04-23T21:22:24.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645551+0000", 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9649999999999999e-06 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124511 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652832+0000", 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0086904040000000005, 2024-04-23T21:22:24.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.016821843, 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124511 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652832+0000", 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652832+0000", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652833+0000", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652835+0000", 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.911e-06 2024-04-23T21:22:24.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652840+0000", 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3210000000000004e-06 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652844+0000", 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5340000000000001e-06 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.253 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124710 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.659836+0000", 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0016868090000000001, 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0098475100000000003, 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124710 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659836+0000", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.659836+0000", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660085+0000", 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000249342 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660088+0000", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1200000000000002e-06 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660096+0000", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9149999999999996e-06 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660102+0000", 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0800000000000002e-06 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.256 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124035 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643754+0000", 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017768519999999999, 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025961689999999999, 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124035 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.257 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643754+0000", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643754+0000", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643755+0000", 2024-04-23T21:22:24.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8999999999999996e-07 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643757+0000", 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643764+0000", 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7789999999999998e-06 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643769+0000", 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4009999999999996e-06 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124139 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644648+0000", 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016874687999999999, 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025112427, 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.260 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124139 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644648+0000", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644648+0000", 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644649+0000", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644651+0000", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.238e-06 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644657+0000", 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6069999999999996e-06 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644662+0000", 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9439999999999996e-06 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124247 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645557+0000", 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015965348000000001, 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024230373999999999, 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124247 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645557+0000", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645557+0000", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645558+0000", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645560+0000", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0530000000000001e-06 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645565+0000", 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.31e-06 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645570+0000", 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8559999999999998e-06 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124324 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647448+0000", 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014074405999999999, 2024-04-23T21:22:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022369185, 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124324 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647448+0000", 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647448+0000", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647449+0000", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1399999999999997e-07 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647451+0000", 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647457+0000", 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8590000000000001e-06 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647462+0000", 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6650000000000002e-06 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124390 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.269 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648694+0000", 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012829175, 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021150643, 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124390 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648694+0000", 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648694+0000", 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648694+0000", 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648696+0000", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9589999999999998e-06 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648701+0000", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2780000000000001e-06 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648706+0000", 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4299999999999999e-06 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.272 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124512 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652851+0000", 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0086713620000000002, 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017022089000000001, 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.273 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124512 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652851+0000", 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652851+0000", 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652852+0000", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652853+0000", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.736e-06 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652859+0000", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6239999999999999e-06 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652864+0000", 2024-04-23T21:22:24.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.707e-06 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124580 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.655977+0000", 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0055461030000000001, 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013923312, 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.276 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124580 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655977+0000", 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655977+0000", 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655977+0000", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.06e-07 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655981+0000", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.473e-06 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655989+0000", 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7249999999999997e-06 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.655995+0000", 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2389999999999999e-06 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124776 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663485+0000", 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9980383, 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0064634669999999997, 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124776 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663485+0000", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663485+0000", 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663485+0000", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9500000000000003e-07 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663492+0000", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6529999999999997e-06 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663502+0000", 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0176999999999999e-05 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663513+0000", 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0749e-05 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124037 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643806+0000", 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.01771673, 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026198822, 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.282 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124037 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643806+0000", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643806+0000", 2024-04-23T21:22:24.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643807+0000", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5799999999999999e-07 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643809+0000", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643818+0000", 2024-04-23T21:22:24.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1759999999999999e-06 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643824+0000", 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6060000000000002e-06 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124151 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644668+0000", 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016854345, 2024-04-23T21:22:24.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025395081, 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124151 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644668+0000", 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644668+0000", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644669+0000", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644671+0000", 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.007e-06 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644688+0000", 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7482e-05 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644694+0000", 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6659999999999996e-06 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124325 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.288 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647469+0000", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014053784999999999, 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022648720000000001, 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124325 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647469+0000", 2024-04-23T21:22:24.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647469+0000", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647469+0000", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647481+0000", 2024-04-23T21:22:24.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1615e-05 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647488+0000", 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6209999999999998e-06 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647493+0000", 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1059999999999999e-06 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.291 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124391 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648715+0000", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012807987, 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021464746999999999, 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124391 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648715+0000", 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648715+0000", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648715+0000", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648717+0000", 2024-04-23T21:22:24.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0200000000000001e-06 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648723+0000", 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6960000000000002e-06 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648729+0000", 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6239999999999999e-06 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.294 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124452 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649753+0000", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011769949, 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020476609, 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124452 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649753+0000", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649753+0000", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649753+0000", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.53e-07 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649755+0000", 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.858e-06 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649761+0000", 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1510000000000001e-06 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649766+0000", 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6500000000000004e-06 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.297 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124587 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656005+0000", 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0055178010000000001, 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014253920999999999, 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124587 2024-04-23T21:22:24.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656005+0000", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656005+0000", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656006+0000", 2024-04-23T21:22:24.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656008+0000", 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2409999999999998e-06 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656014+0000", 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656020+0000", 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3199999999999999e-06 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124042 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643842+0000", 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017680522000000001, 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026451952000000001, 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.301 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124042 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643842+0000", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643842+0000", 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643843+0000", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643853+0000", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8810000000000008e-06 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643861+0000", 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7309999999999999e-06 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643867+0000", 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.896e-06 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.670282+0000", 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026414930999999999 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124152 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.304 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644703+0000", 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016819486000000002, 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025634558000000002, 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124152 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644703+0000", 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644703+0000", 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644704+0000", 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644706+0000", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3470000000000001e-06 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644712+0000", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2960000000000004e-06 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644717+0000", 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0300000000000001e-06 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124326 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647500+0000", 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.014022995999999999, 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022875355999999999, 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124326 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647500+0000", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647500+0000", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647500+0000", 2024-04-23T21:22:24.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647502+0000", 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647509+0000", 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5200000000000003e-06 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647514+0000", 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0010000000000001e-06 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124453 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649771+0000", 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011751360000000001, 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020631170000000001, 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.311 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124453 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649771+0000", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649771+0000", 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649772+0000", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4899999999999995e-07 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649774+0000", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0030000000000001e-06 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649779+0000", 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5289999999999999e-06 2024-04-23T21:22:24.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649784+0000", 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6199999999999998e-06 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124588 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656026+0000", 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0054964239999999998, 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014404375000000001, 2024-04-23T21:22:24.314 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124588 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656026+0000", 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656026+0000", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656027+0000", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6400000000000003e-07 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656029+0000", 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1210000000000001e-06 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656035+0000", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4749999999999998e-06 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656040+0000", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7859999999999996e-06 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124778 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663557+0000", 2024-04-23T21:22:24.317 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9979663, 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0069013089999999996, 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124778 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663557+0000", 2024-04-23T21:22:24.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663557+0000", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663557+0000", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.06e-07 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663560+0000", 2024-04-23T21:22:24.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4849999999999999e-06 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663567+0000", 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1459999999999999e-06 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663572+0000", 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2830000000000001e-06 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124044 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.643903+0000", 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.017619623000000001, 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026588187999999999, 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124044 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643903+0000", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643903+0000", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643904+0000", 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643906+0000", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.678e-06 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643914+0000", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0709999999999995e-06 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.643920+0000", 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7910000000000001e-06 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.670480+0000", 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026559805999999998 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124158 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644724+0000", 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016798758, 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025816800000000001, 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.324 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124158 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644724+0000", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644724+0000", 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644725+0000", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9999999999999998e-07 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644727+0000", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2050000000000001e-06 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644733+0000", 2024-04-23T21:22:24.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5899999999999996e-06 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644738+0000", 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.147e-06 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124260 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645650+0000", 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015873188999999999, 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024921022000000001, 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.327 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124260 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645650+0000", 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645650+0000", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645650+0000", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5600000000000001e-07 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645652+0000", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2180000000000002e-06 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645658+0000", 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0859999999999998e-06 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645664+0000", 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1730000000000003e-06 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124456 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649790+0000", 2024-04-23T21:22:24.330 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011733217000000001, 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020807784999999999, 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124456 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649790+0000", 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649790+0000", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649790+0000", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649792+0000", 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9379999999999999e-06 2024-04-23T21:22:24.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649797+0000", 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5879999999999997e-06 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649802+0000", 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3239999999999996e-06 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.333 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124522 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652911+0000", 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0086113149999999996, 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.017740446, 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124522 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652911+0000", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652911+0000", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652912+0000", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3999999999999997e-07 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652914+0000", 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.922e-06 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652919+0000", 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6590000000000004e-06 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652930+0000", 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0346999999999999e-05 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124651 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658693+0000", 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0028294919999999999, 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012016183999999999, 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124651 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658693+0000", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658693+0000", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658694+0000", 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1600000000000001e-07 2024-04-23T21:22:24.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658698+0000", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9049999999999999e-06 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658707+0000", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8780000000000004e-06 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658713+0000", 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5830000000000003e-06 2024-04-23T21:22:24.339 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124780 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663579+0000", 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9979439, 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0071882459999999997, 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.340 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124780 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663579+0000", 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663579+0000", 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663579+0000", 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663582+0000", 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.182e-06 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663588+0000", 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9089999999999997e-06 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663592+0000", 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6290000000000001e-06 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124159 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644745+0000", 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016777924999999999, 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026080137999999999, 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.343 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124159 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644745+0000", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644745+0000", 2024-04-23T21:22:24.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644745+0000", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000002e-07 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644748+0000", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1780000000000002e-06 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644754+0000", 2024-04-23T21:22:24.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1820000000000003e-06 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644759+0000", 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9119999999999997e-06 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124261 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645671+0000", 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015852224000000002, 2024-04-23T21:22:24.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025215192000000001, 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124261 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645671+0000", 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645671+0000", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645671+0000", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645673+0000", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.137e-06 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645679+0000", 2024-04-23T21:22:24.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1789999999999996e-06 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645684+0000", 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6720000000000003e-06 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124396 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648771+0000", 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012751503000000001, 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022142352000000001, 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.349 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124396 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648771+0000", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648771+0000", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648772+0000", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.53e-07 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648773+0000", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8020000000000001e-06 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648779+0000", 2024-04-23T21:22:24.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7780000000000004e-06 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648784+0000", 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3320000000000002e-06 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124457 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.649807+0000", 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.011715579, 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.021145270000000001, 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.352 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124457 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649807+0000", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.649807+0000", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651423+0000", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0016159830000000001 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651427+0000", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-06 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651436+0000", 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0909999999999999e-06 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651443+0000", 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.759e-06 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124524 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652936+0000", 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0085868520000000007, 2024-04-23T21:22:24.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018043739, 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124524 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652936+0000", 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652936+0000", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652936+0000", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5999999999999999e-07 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652938+0000", 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0789999999999999e-06 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652944+0000", 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.536e-06 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652949+0000", 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5990000000000004e-06 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.358 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124590 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656070+0000", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0054524830000000002, 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.014946151, 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124590 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656070+0000", 2024-04-23T21:22:24.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656070+0000", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656071+0000", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.39e-07 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656073+0000", 2024-04-23T21:22:24.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.08e-06 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656079+0000", 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1240000000000002e-06 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656084+0000", 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1189999999999996e-06 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.361 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124654 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658726+0000", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0027971210000000001, 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012319754, 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124654 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658726+0000", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658726+0000", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658726+0000", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1200000000000003e-07 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658728+0000", 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658734+0000", 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9689999999999999e-06 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658739+0000", 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.831e-06 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124721 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.660199+0000", 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0013240389999999999, 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010877727, 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124721 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660199+0000", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660199+0000", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660931+0000", 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00073209900000000005 2024-04-23T21:22:24.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660935+0000", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0790000000000002e-06 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660944+0000", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0580000000000004e-06 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660951+0000", 2024-04-23T21:22:24.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.815e-06 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124160 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644766+0000", 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016756605000000001, 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026378041000000001, 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.368 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124160 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644766+0000", 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644766+0000", 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644767+0000", 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644769+0000", 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2120000000000002e-06 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644776+0000", 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1199999999999996e-06 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644781+0000", 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3900000000000001e-06 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124268 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645690+0000", 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015832664, 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025511053999999998, 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124268 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645690+0000", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645690+0000", 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645690+0000", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645693+0000", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1220000000000002e-06 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645700+0000", 2024-04-23T21:22:24.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999997e-06 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645706+0000", 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2630000000000003e-06 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124336 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647655+0000", 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013867312, 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023602933, 2024-04-23T21:22:24.374 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124336 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647655+0000", 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647655+0000", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647656+0000", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3e-07 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647660+0000", 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6950000000000002e-06 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647667+0000", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3969999999999997e-06 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647674+0000", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124397 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.648789+0000", 2024-04-23T21:22:24.377 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.012733713000000001, 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022522001, 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124397 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648789+0000", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648789+0000", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648789+0000", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4200000000000002e-07 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648796+0000", 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2500000000000003e-06 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648805+0000", 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5200000000000003e-06 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.648810+0000", 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9520000000000002e-06 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124529 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652955+0000", 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0085676859999999997, 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018410497000000001, 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124529 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652955+0000", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652955+0000", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652955+0000", 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652958+0000", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652963+0000", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652968+0000", 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7040000000000002e-06 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.383 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124661 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658746+0000", 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0027771710000000002, 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012647764000000001, 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124661 2024-04-23T21:22:24.384 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658746+0000", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658746+0000", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658746+0000", 2024-04-23T21:22:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658748+0000", 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0370000000000001e-06 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658754+0000", 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7660000000000004e-06 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.386 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658758+0000", 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.476e-06 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124722 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.660963+0000", 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00055986399999999998, 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010459016999999999, 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.387 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124722 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660963+0000", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660963+0000", 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.388 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660963+0000", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2200000000000004e-07 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660966+0000", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.779e-06 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660973+0000", 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3500000000000002e-06 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.389 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660978+0000", 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0649999999999998e-06 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124166 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644788+0000", 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016734345000000001, 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026672821999999999, 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.390 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124166 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644788+0000", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.391 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644788+0000", 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644789+0000", 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.39e-07 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644799+0000", 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.039e-05 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.392 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644806+0000", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.951e-06 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644812+0000", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5770000000000001e-06 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124269 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.645712+0000", 2024-04-23T21:22:24.393 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.015810785000000001, 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025813764999999999, 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124269 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645712+0000", 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.394 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645712+0000", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645712+0000", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645714+0000", 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.024e-06 2024-04-23T21:22:24.395 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645720+0000", 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0549999999999996e-06 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.645725+0000", 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6829999999999999e-06 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.396 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124337 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647685+0000", 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013837721000000001, 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.023895357999999998, 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124337 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.397 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647685+0000", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647685+0000", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647686+0000", 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.63e-07 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.398 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647688+0000", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1380000000000001e-06 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647693+0000", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7710000000000003e-06 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647698+0000", 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.848e-06 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.399 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124531 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.652973+0000", 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0085494149999999994, 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018663448999999999, 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124531 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.400 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652973+0000", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652973+0000", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652974+0000", 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:22:24.401 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652975+0000", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8080000000000001e-06 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652981+0000", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4240000000000001e-06 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.652985+0000", 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5290000000000002e-06 2024-04-23T21:22:24.402 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124664 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658765+0000", 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002757823, 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.012925028999999999, 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.403 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124664 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658765+0000", 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658765+0000", 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.404 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658765+0000", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1199999999999999e-07 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658767+0000", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0109999999999999e-06 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658773+0000", 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6300000000000003e-06 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.405 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658777+0000", 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4560000000000002e-06 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124723 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.660985+0000", 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00053812900000000004, 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010759259, 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.406 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124723 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660985+0000", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.407 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660985+0000", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660985+0000", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3500000000000002e-07 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660987+0000", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2290000000000002e-06 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.408 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660993+0000", 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1829999999999999e-06 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.660998+0000", 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7029999999999997e-06 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124784 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.663652+0000", 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9978709, 2024-04-23T21:22:24.409 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0081219529999999995, 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124784 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663652+0000", 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.410 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663652+0000", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.663652+0000", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8700000000000001e-07 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664042+0000", 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00038963099999999998 2024-04-23T21:22:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664049+0000", 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5000000000000002e-06 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664055+0000", 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7060000000000001e-06 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.412 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124167 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644822+0000", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016700515999999999, 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026984649999999999, 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124167 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.413 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644822+0000", 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644822+0000", 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644823+0000", 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0500000000000003e-07 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.414 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644835+0000", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1931e-05 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644842+0000", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0029999999999997e-06 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644847+0000", 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.502e-06 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.415 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124339 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.647706+0000", 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.013817228000000001, 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.024152941000000001, 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124339 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.416 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647706+0000", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647706+0000", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647706+0000", 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.65e-07 2024-04-23T21:22:24.417 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647708+0000", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.03e-06 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647714+0000", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7799999999999997e-06 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.647718+0000", 2024-04-23T21:22:24.418 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6820000000000002e-06 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124666 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658784+0000", 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0027391999999999998, 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013132338, 2024-04-23T21:22:24.419 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124666 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658784+0000", 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.420 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658784+0000", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658784+0000", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658786+0000", 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2579999999999998e-06 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.421 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658793+0000", 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3790000000000003e-06 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658797+0000", 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7759999999999997e-06 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.422 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124724 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661004+0000", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00051831000000000002, 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.010969309, 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124724 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661004+0000", 2024-04-23T21:22:24.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661004+0000", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661005+0000", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8200000000000001e-07 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661007+0000", 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:24.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661014+0000", 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8469999999999998e-06 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661018+0000", 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6709999999999998e-06 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124174 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.425 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.644873+0000", 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.016649817000000001, 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027155144999999999, 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124174 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644873+0000", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644873+0000", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644874+0000", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9100000000000004e-07 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644876+0000", 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6800000000000002e-06 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644885+0000", 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8689999999999993e-06 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.644891+0000", 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8209999999999998e-06 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124465 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.651514+0000", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.010008830999999999, 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.020571434999999999, 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124465 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651514+0000", 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651514+0000", 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651514+0000", 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651516+0000", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651526+0000", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.38e-06 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.651531+0000", 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0579999999999997e-06 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.431 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124599 3.1 3.68720c21 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.656168+0000", 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0053550209999999997, 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.015969476, 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124599 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.432 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656168+0000", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656168+0000", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656168+0000", 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:24.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656170+0000", 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9860000000000001e-06 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656176+0000", 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3740000000000003e-06 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.656181+0000", 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9439999999999996e-06 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124667 3.1 3.82cbb6e1 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.658805+0000", 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.002718158, 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013390542, 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.435 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124667 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658805+0000", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658805+0000", 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658805+0000", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0299999999999999e-07 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658807+0000", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0770000000000002e-06 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658813+0000", 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8089999999999998e-06 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.658818+0000", 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124725 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.661039+0000", 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.000483559, 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011210711999999999, 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.438 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124725 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661039+0000", 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661039+0000", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661040+0000", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9299999999999998e-07 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661042+0000", 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2960000000000001e-06 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661049+0000", 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5339999999999996e-06 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.661054+0000", 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0930000000000002e-06 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:124786 3.9 3.7a516269 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e79)", 2024-04-23T21:22:24.441 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:23.664091+0000", 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9974313, 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0082168329999999998, 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "queued for pg", 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 124786 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664091+0000", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664091+0000", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664092+0000", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.01e-07 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664094+0000", 2024-04-23T21:22:24.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2139999999999999e-06 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664100+0000", 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:23.664105+0000", 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9919999999999998e-06 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.444 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:24.445 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 224 2024-04-23T21:22:24.445 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:24.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:24.447 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:24.446+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4879 write 0x95abc34 thru 0x95ae4c1 (0x288e bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4881 punch from 0x455d235 to 0x456346c, (0x6237 bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4883 write 0x37d6556 thru 0x37e431a (0xddc5 bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4884 punch from 0x8e560c1 to 0x8e65214, (0xf153 bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4885 read 0xa01a67e thru 0xa01f424 (0x4da7 bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4886 trunc from 0xebcf052 to 0xb64a136 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4888 trunc from 0xb64a136 to 0x5c5e58d 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4889 read 0x13c1e1d thru 0x13cbd30 (0x9f14 bytes) 2024-04-23T21:22:24.454 INFO:teuthology.orchestra.run.smithi111.stdout:4891 write 0xeccd14c thru 0xecda8db (0xd790 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4892 writesame 0x9b274ee thru 0x9b303dd data_size 0x8(0x8ef0 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4894 write 0x8232153 thru 0x82399a8 (0x7856 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4895 trunc from 0xecda8dc to 0xdf66668 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4896 read 0xa49d2f7 thru 0xa4a31b8 (0x5ec2 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4898 punch from 0xb4a2f87 to 0xb4b1d62, (0xeddb bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4900 read 0x72e0f30 thru 0x72e4f23 (0x3ff4 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4903 read 0x2e97d1b thru 0x2ea0e35 (0x911b bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4906 write 0xdb825c thru 0xdc4e10 (0xcbb5 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4907 trunc from 0xdf66668 to 0x650cd51 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4908 trunc from 0x650cd51 to 0xdc526a5 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4910 trunc from 0xdc526a5 to 0xcb24fff 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4912 writesame 0x9bca284 thru 0x9bd18e4 data_size 0x13(0x7661 bytes) 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.455 INFO:teuthology.orchestra.run.smithi111.stdout:4915 write 0xc620e56 thru 0xc62fa48 (0xebf3 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4916 write 0xdf645c2 thru 0xdf64c9b (0x6da bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4918 trunc from 0xdf64c9c to 0x52a31ff 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4919 trunc from 0x52a31ff to 0x217690a 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4920 read 0x1524303 thru 0x1532386 (0xe084 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4921 writesame 0x9078ff7 thru 0x9085f71 data_size 0x5(0xcf7b bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4922 trunc from 0x9085f72 to 0xc4f0544 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4923 trunc from 0xc4f0544 to 0xe7fa799 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4924 write 0x3d3a27a thru 0x3d3cc10 (0x2997 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4926 punch from 0x898b941 to 0x898c556, (0xc15 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4927 read 0xd042d35 thru 0xd04e4e9 (0xb7b5 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4928 trunc from 0xe7fa799 to 0x2bfb718 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4929 write 0x2260b80 thru 0x2265544 (0x49c5 bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4931 trunc from 0x2bfb718 to 0x478091d 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4932 write 0x2f2d0e5 thru 0x2f373ae (0xa2ca bytes) 2024-04-23T21:22:24.456 INFO:teuthology.orchestra.run.smithi111.stdout:4933 punch from 0x20428f0 to 0x204a2a4, (0x79b4 bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4934 read 0x3e5aec8 thru 0x3e60839 (0x5972 bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4935 writesame 0x7b38ccc thru 0x7b3f837 data_size 0x1(0x6b6c bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4936 punch from 0x5c5ef23 to 0x5c5fe90, (0xf6d bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4938 punch from 0x6aad189 to 0x6ab23f8, (0x526f bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4939 write 0x80870b1 thru 0x8094c8b (0xdbdb bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4940 write 0xc31beb1 thru 0xc32626a (0xa3ba bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4942 writesame 0xd99ba86 thru 0xd9a6612 data_size 0x1(0xab8d bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4946 read 0x111dba5 thru 0x1123892 (0x5cee bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4947 read 0x7e327fd thru 0x7e35cf5 (0x34f9 bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4948 trunc from 0xd9a6613 to 0x71999f 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4949 punch from 0x4e9c15 to 0x4ee1f0, (0x45db bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4950 trunc from 0x71999f to 0x68273bf 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4951 trunc from 0x68273bf to 0x37e155b 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4952 read 0xfed5a6 thru 0xffa008 (0xca63 bytes) 2024-04-23T21:22:24.457 INFO:teuthology.orchestra.run.smithi111.stdout:4953 read 0x2faf72a thru 0x2fb7545 (0x7e1c bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4954 writesame 0x23aabda thru 0x23ad397 data_size 0x1(0x27be bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4955 read 0x376f0b3 thru 0x377e91a (0xf868 bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4956 write 0xabb2b6a thru 0xabbde12 (0xb2a9 bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4957 read 0x5161a17 thru 0x516c100 (0xa6ea bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4958 trunc from 0xabbde13 to 0xade0dc7 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4959 trunc from 0xade0dc7 to 0x6c6a330 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4960 trunc from 0x6c6a330 to 0x4485c78 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4962 trunc from 0x4485c78 to 0x5265fab 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4964 write 0xddd38b0 thru 0xdddebca (0xb31b bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4966 write 0x2dedbe7 thru 0x2df1009 (0x3423 bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4967 write 0xdde1ebd thru 0xddedfbb (0xc0ff bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4968 write 0x6a9265a thru 0x6a94a10 (0x23b7 bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4971 punch from 0xa1c5c27 to 0xa1ca0da, (0x44b3 bytes) 2024-04-23T21:22:24.458 INFO:teuthology.orchestra.run.smithi111.stdout:4972 writesame 0x289b626 thru 0x28a93e0 data_size 0x3f(0xddbb bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4973 write 0xcf91820 thru 0xcf980d6 (0x68b7 bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4974 read 0xd3c20b1 thru 0xd3ca3ff (0x834f bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4975 write 0x16131fb thru 0x16138ef (0x6f5 bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4976 trunc from 0xddedfbc to 0xb25e3cc 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4977 read 0xa631b97 thru 0xa635579 (0x39e3 bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stdout:4980 writesame 0xd2c0079 thru 0xd2cdc9b data_size 0x55(0xdc23 bytes) 2024-04-23T21:22:24.459 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T21:22:24.460 INFO:teuthology.orchestra.run.smithi111.stdout:4984 writesame 0xda16171 thru 0xda165f4 data_size 0x4(0x484 bytes) 2024-04-23T21:22:24.460 INFO:teuthology.orchestra.run.smithi111.stdout:4986 trunc from 0xda165f5 to 0x87bb0d7 2024-04-23T21:22:24.461 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:4989 writesame 0x2669b95 thru 0x26789b7 data_size 0x7(0xee23 bytes) 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:4994 trunc from 0x87bb0d7 to 0x87040e9 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:4996 read 0x5b42a6 thru 0x5c1360 (0xd0bb bytes) 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:4997 read 0x4d65cbb thru 0x4d6cb8c (0x6ed2 bytes) 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:5000 trunc from 0x87040e9 to 0xc36c4f9 2024-04-23T21:22:24.465 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5002 read 0x9d50d57 thru 0x9d52761 (0x1a0b bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5003 read 0xc14927a thru 0xc14b0b1 (0x1e38 bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5004 punch from 0x429a3e7 to 0x429e34c, (0x3f65 bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5007 read 0x8dfa523 thru 0x8dfcba8 (0x2686 bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5008 read 0x2d95b66 thru 0x2da0dbb (0xb256 bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5010 trunc from 0xc36c4f9 to 0xcf46e8a 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5011 write 0x1cddd75 thru 0x1ce518f (0x741b bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5012 trunc from 0xcf46e8a to 0xbd60404 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5013 trunc from 0xbd60404 to 0x31ca67 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5015 read 0x2cfbc1 thru 0x2d253c (0x297c bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5016 read 0xda03d thru 0xe9485 (0xf449 bytes) 2024-04-23T21:22:24.466 INFO:teuthology.orchestra.run.smithi111.stdout:5019 read 0xbddf2 thru 0xc80de (0xa2ed bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5020 trunc from 0x31ca67 to 0x9ae9790 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5025 punch from 0x4005cb5 to 0x400fa9f, (0x9dea bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5026 writesame 0x24676dd thru 0x246cc40 data_size 0xa(0x5564 bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5027 read 0x8fb88 thru 0x98106 (0x857f bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5029 write 0xe355c2 thru 0xe36113 (0xb52 bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5030 read 0x4801b6f thru 0x480343a (0x18cc bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5032 read 0x2e4f2da thru 0x2e5e2f5 (0xf01c bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5033 read 0x1cf9c63 thru 0x1d04f4b (0xb2e9 bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5035 write 0x2351aa9 thru 0x235a897 (0x8def bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5036 write 0x45a86a6 thru 0x45b4781 (0xc0dc bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5037 read 0x20e5533 thru 0x20eec1f (0x96ed bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5038 write 0x313ea59 thru 0x3141aa1 (0x3049 bytes) 2024-04-23T21:22:24.467 INFO:teuthology.orchestra.run.smithi111.stdout:5039 read 0x6676d58 thru 0x667af90 (0x4239 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5040 punch from 0x8ab0155 to 0x8abce41, (0xccec bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5041 read 0x480bda thru 0x488e41 (0x8268 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5042 write 0x25e1633 thru 0x25e5bc1 (0x458f bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5043 read 0x6157500 thru 0x6165d67 (0xe868 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5044 trunc from 0x9ae9790 to 0xb156578 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5045 write 0x33e0c38 thru 0x33e1124 (0x4ed bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5047 read 0x1114e2e thru 0x111a80f (0x59e2 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5048 write 0x66fdf26 thru 0x6705b5f (0x7c3a bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5049 write 0x50346a2 thru 0x50422a6 (0xdc05 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5050 read 0x5ed6901 thru 0x5ee16f0 (0xadf0 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5051 read 0x65104b2 thru 0x65187b4 (0x8303 bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5053 write 0x9349853 thru 0x934cb2c (0x32da bytes) 2024-04-23T21:22:24.468 INFO:teuthology.orchestra.run.smithi111.stdout:5054 trunc from 0xb156578 to 0x6cc6bb7 2024-04-23T21:22:24.478 INFO:teuthology.orchestra.run.smithi111.stdout:5055 trunc from 0x6cc6bb7 to 0xb73cb22 2024-04-23T21:22:24.484 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.492 INFO:teuthology.orchestra.run.smithi111.stdout:5057 write 0xd7368b8 thru 0xd73ea27 (0x8170 bytes) 2024-04-23T21:22:24.495 INFO:teuthology.orchestra.run.smithi111.stdout:5058 punch from 0x3d53e09 to 0x3d57b70, (0x3d67 bytes) 2024-04-23T21:22:24.496 INFO:teuthology.orchestra.run.smithi111.stdout:5059 read 0x97559ab thru 0x9756281 (0x8d7 bytes) 2024-04-23T21:22:24.496 INFO:teuthology.orchestra.run.smithi111.stdout:5060 write 0x2906238 thru 0x290edc4 (0x8b8d bytes) 2024-04-23T21:22:24.498 INFO:teuthology.orchestra.run.smithi111.stdout:5061 read 0x33f3650 thru 0x33f8e6f (0x5820 bytes) 2024-04-23T21:22:24.499 INFO:teuthology.orchestra.run.smithi111.stdout:5062 read 0x5f9d12d thru 0x5fac736 (0xf60a bytes) 2024-04-23T21:22:24.499 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.499 INFO:teuthology.orchestra.run.smithi111.stdout:5066 write 0x693914d thru 0x693e14b (0x4fff bytes) 2024-04-23T21:22:24.501 INFO:teuthology.orchestra.run.smithi111.stdout:5067 write 0x3133cce thru 0x313f2c7 (0xb5fa bytes) 2024-04-23T21:22:24.503 INFO:teuthology.orchestra.run.smithi111.stdout:5068 write 0x75cc7a9 thru 0x75d9b6c (0xd3c4 bytes) 2024-04-23T21:22:24.506 INFO:teuthology.orchestra.run.smithi111.stdout:5070 read 0x2c63bd9 thru 0x2c6fc85 (0xc0ad bytes) 2024-04-23T21:22:24.506 INFO:teuthology.orchestra.run.smithi111.stdout:5071 punch from 0x54ace00 to 0x54bcde7, (0xffe7 bytes) 2024-04-23T21:22:24.507 INFO:teuthology.orchestra.run.smithi111.stdout:5072 write 0x5498c90 thru 0x549ae13 (0x2184 bytes) 2024-04-23T21:22:24.518 INFO:teuthology.orchestra.run.smithi111.stdout:5073 write 0xee53c24 thru 0xee5e672 (0xaa4f bytes) 2024-04-23T21:22:24.521 INFO:teuthology.orchestra.run.smithi111.stdout:5074 write 0x7eab83f thru 0x7eae7db (0x2f9d bytes) 2024-04-23T21:22:24.524 INFO:teuthology.orchestra.run.smithi111.stdout:5075 writesame 0x16ed9fa thru 0x16f6c87 data_size 0xd(0x928e bytes) 2024-04-23T21:22:24.526 INFO:teuthology.orchestra.run.smithi111.stdout:5076 punch from 0x728be07 to 0x728e454, (0x264d bytes) 2024-04-23T21:22:24.527 INFO:teuthology.orchestra.run.smithi111.stdout:5077 read 0xec01be9 thru 0xec11b04 (0xff1c bytes) 2024-04-23T21:22:24.527 INFO:teuthology.orchestra.run.smithi111.stdout:5078 read 0x4f96b thru 0x5bf05 (0xc59b bytes) 2024-04-23T21:22:24.528 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.528 INFO:teuthology.orchestra.run.smithi111.stdout:5080 punch from 0xb552b5c to 0xb5545f0, (0x1a94 bytes) 2024-04-23T21:22:24.528 INFO:teuthology.orchestra.run.smithi111.stdout:5081 punch from 0x8e0cb62 to 0x8e113c3, (0x4861 bytes) 2024-04-23T21:22:24.528 INFO:teuthology.orchestra.run.smithi111.stdout:5082 write 0x14c436e thru 0x14d3c81 (0xf914 bytes) 2024-04-23T21:22:24.530 INFO:teuthology.orchestra.run.smithi111.stdout:5083 writesame 0x953d165 thru 0x954ba85 data_size 0x1(0xe921 bytes) 2024-04-23T21:22:24.547 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:24.548+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:24.571 INFO:teuthology.orchestra.run.smithi111.stdout:5084 trunc from 0xee5e673 to 0xa1740a 2024-04-23T21:22:24.591 INFO:teuthology.orchestra.run.smithi111.stdout:5085 punch from 0x7483f9 to 0x74c554, (0x415b bytes) 2024-04-23T21:22:24.592 INFO:teuthology.orchestra.run.smithi111.stdout:5086 read 0x629abe thru 0x62eb59 (0x509c bytes) 2024-04-23T21:22:24.592 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.592 INFO:teuthology.orchestra.run.smithi111.stdout:5088 trunc from 0xa1740a to 0x2c6b46f 2024-04-23T21:22:24.595 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.595 INFO:teuthology.orchestra.run.smithi111.stdout:5090 read 0x2642b5 thru 0x26e3fe (0xa14a bytes) 2024-04-23T21:22:24.598 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:24.598 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:24.598 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:24.598 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:24.599 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:24.607 INFO:teuthology.orchestra.run.smithi111.stdout:5091 write 0x5c7684f thru 0x5c7acbe (0x4470 bytes) 2024-04-23T21:22:24.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:24.625 INFO:teuthology.orchestra.run.smithi111.stdout:5092 write 0xd82db15 thru 0xd8375ed (0x9ad9 bytes) 2024-04-23T21:22:24.628 INFO:teuthology.orchestra.run.smithi111.stdout:5093 write 0x6231d85 thru 0x6235ce1 (0x3f5d bytes) 2024-04-23T21:22:24.643 INFO:teuthology.orchestra.run.smithi111.stdout:5094 trunc from 0xd8375ee to 0x32a087d 2024-04-23T21:22:24.647 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:24.648+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:24.655 INFO:teuthology.orchestra.run.smithi111.stdout:5095 read 0x14870d0 thru 0x148c6b7 (0x55e8 bytes) 2024-04-23T21:22:24.671 INFO:teuthology.orchestra.run.smithi111.stdout:5096 writesame 0xafe52a2 thru 0xaff2251 data_size 0x8(0xcfb0 bytes) 2024-04-23T21:22:24.671 INFO:teuthology.orchestra.run.smithi111.stdout:5097 read 0x3a39417 thru 0x3a3fee8 (0x6ad2 bytes) 2024-04-23T21:22:24.672 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.672 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.672 INFO:teuthology.orchestra.run.smithi111.stdout:5102 read 0x15f54a9 thru 0x15f9d4b (0x48a3 bytes) 2024-04-23T21:22:24.672 INFO:teuthology.orchestra.run.smithi111.stdout:5105 punch from 0x914e275 to 0x915c2d7, (0xe062 bytes) 2024-04-23T21:22:24.672 INFO:teuthology.orchestra.run.smithi111.stdout:5106 write 0x7f1f3dc thru 0x7f229ea (0x360f bytes) 2024-04-23T21:22:24.677 INFO:teuthology.orchestra.run.smithi111.stdout:5108 write 0x5a4741a thru 0x5a4a99d (0x3584 bytes) 2024-04-23T21:22:24.680 INFO:teuthology.orchestra.run.smithi111.stdout:5111 writesame 0x64b54a3 thru 0x64bc5f0 data_size 0x1(0x714e bytes) 2024-04-23T21:22:24.703 INFO:teuthology.orchestra.run.smithi111.stdout:5112 write 0xcbdf9ce thru 0xcbe2615 (0x2c48 bytes) 2024-04-23T21:22:24.707 INFO:teuthology.orchestra.run.smithi111.stdout:5113 writesame 0x75ac27d thru 0x75af4c9 data_size 0x1(0x324d bytes) 2024-04-23T21:22:24.714 INFO:teuthology.orchestra.run.smithi111.stdout:5114 write 0x9853341 thru 0x9862958 (0xf618 bytes) 2024-04-23T21:22:24.718 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.718 INFO:teuthology.orchestra.run.smithi111.stdout:5116 write 0x8132dc9 thru 0x813ccc8 (0x9f00 bytes) 2024-04-23T21:22:24.721 INFO:teuthology.orchestra.run.smithi111.stdout:5117 writesame 0xb49cf9 thru 0xb51299 data_size 0x1(0x75a1 bytes) 2024-04-23T21:22:24.734 INFO:teuthology.orchestra.run.smithi111.stdout:5118 read 0x8efaa06 thru 0x8f08aad (0xe0a8 bytes) 2024-04-23T21:22:24.734 INFO:teuthology.orchestra.run.smithi111.stdout:5119 write 0x9fdffc3 thru 0x9fe185a (0x1898 bytes) 2024-04-23T21:22:24.736 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:24.748 INFO:teuthology.orchestra.run.smithi111.stdout:5121 trunc from 0xcbe2616 to 0x35baf02 2024-04-23T21:22:24.748 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:24.749+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:24.759 INFO:teuthology.orchestra.run.smithi111.stdout:5125 read 0x30cb9cf thru 0x30d2725 (0x6d57 bytes) 2024-04-23T21:22:24.759 INFO:teuthology.orchestra.run.smithi111.stdout:5126 read 0x1053d80 thru 0x105bf65 (0x81e6 bytes) 2024-04-23T21:22:24.762 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806634133000003, 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.763 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806596012999997, 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.767 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:24.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:24.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806573614000001, 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:24.771 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.775 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806554218000002, 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:24.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:24.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:24.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.779 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806534620000001, 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.783 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806514407000002, 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.784 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:24.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:24.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:24.787 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806482240999998, 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.788 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806457627999997, 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.792 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.793 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.794 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.795 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.80643663, 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.796 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.797 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.798 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.799 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806416331000001, 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.800 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:24.801 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.802 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:24.803 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.806338541999999, 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.804 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:24.805 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.806 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.807 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.804843167999998, 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:24.808 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.809 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.810 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.811 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:24.812 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.804793277999998, 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.813 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:24.814 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:24.815 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.816 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.804769509000003, 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.817 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.818 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.819 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.820 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.804748502999999, 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:24.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:24.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:24.823 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.824 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.804728898, 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.788726748999999, 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:24.828 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:24.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.788708237999998, 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:24.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:24.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:24.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 37.788687793000001, 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:24.836 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:24.840 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.809610581000001, 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:24.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:24.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:24.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.845 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:24.846 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:24.846 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:24.847 INFO:teuthology.orchestra.run.smithi111.stdout:5127 write 0x517fafe thru 0x5183bc9 (0x40cc bytes) 2024-04-23T21:22:24.847 INFO:teuthology.orchestra.run.smithi111.stdout:5129 trunc from 0x5183bca to 0x1702fc5 2024-04-23T21:22:24.847 INFO:teuthology.orchestra.run.smithi111.stdout:5130 clone 43 order 22 su 32768 sc 4 2024-04-23T21:22:24.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:24.849 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:24.849+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:24.950 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:24.951+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:24.991 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:24.991 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:24.991 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:24.999 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:25.083 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:25.084+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:25.109 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:25.142 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:25.142 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:25.142 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:25.142 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:25.142 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:25.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:25.191 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:25.192+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:25.230 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:25.231 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:25.248 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:25.248 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:25.252 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:25.253+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:25.264 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:25.264 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:25.281 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:25.281 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.4200040349999998, 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:25.296 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:25.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:25.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3535855259999998, 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:25.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:25.304 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3535681449999997, 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.305 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3535505880000001, 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.309 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:25.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:25.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:25.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3518903929999997, 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:25.313 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3518684969999999, 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:25.317 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:25.321 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.351841147, 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.325 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.351817059, 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.326 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:25.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.329 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3517859999999997, 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:25.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:25.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:25.333 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3505859319999995, 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.334 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:25.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.337 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3505634989999997, 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:25.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:25.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3505399110000003, 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.342 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:25.346 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3505199719999998, 2024-04-23T21:22:25.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:25.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:25.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3504914970000002, 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:25.354 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.356 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.358 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3491087530000003, 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.360 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.362 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3440121290000002, 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:25.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:25.366 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3436655030000004, 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.367 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.369 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.370 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3321505990000002, 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.371 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.373 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.374 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3321306599999998, 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.375 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:25.376 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:25.377 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.378 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.3308303889999999, 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:25.379 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:25.380 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:25.381 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:25.382 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:25.383 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:25.383 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:22:25.384 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:25.385 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:25.386 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:25.386 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:25.386 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:25.387 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:25.385+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:25.387 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T21:22:25.388 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-23T21:22:25.388 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:25.388 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:25.389 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:25.389 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:25.417 INFO:teuthology.orchestra.run.smithi111.stdout:leaving image image_client.0-clone42 intact 2024-04-23T21:22:25.486 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:25.486+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:25.535 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:25.535 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:25.544 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:25.553 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #41, image image_client.0-clone41 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent42 2024-04-23T21:22:25.557 INFO:teuthology.orchestra.run.smithi111.stdout:5131 punch from 0x4487ff to 0x44c7ca, (0x3fcb bytes) 2024-04-23T21:22:25.579 INFO:teuthology.orchestra.run.smithi111.stdout:5132 writesame 0x9605b4d thru 0x960c9d3 data_size 0x1(0x6e87 bytes) 2024-04-23T21:22:25.579 INFO:teuthology.orchestra.run.smithi111.stdout:5133 clone 44 order 20 su 262144 sc 10 2024-04-23T21:22:25.586 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:25.586+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:25.687 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:25.688+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:25.688 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:25.688 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:25.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:25.788 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:25.788+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:25.838 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:25.839 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:25.847 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:25.889 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:25.889+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:25.989 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:25.989 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:25.990 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:25.990+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:25.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:26.090 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:26.091+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:26.101 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.101 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.109 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:26.192 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:26.192+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:26.253 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.253 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:26.293 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:26.293+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:26.325 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:22:26.329 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T21:22:26.347 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-23T21:22:26.386 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.387 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.394 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:26.394+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:26.396 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:26.435 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone43 from 0x960c9d4 (overlap 0x1702fc5) to 0xdce976 2024-04-23T21:22:26.495 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:26.495+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:26.558 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.558 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:26.596 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:26.597+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:26.645 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #42, image image_client.0-clone42 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent43 2024-04-23T21:22:26.676 INFO:teuthology.orchestra.run.smithi111.stdout:5134 trunc from 0x960c9d4 to 0x9bfdb3f 2024-04-23T21:22:26.678 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.678 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.685 INFO:teuthology.orchestra.run.smithi111.stdout:5135 punch from 0x92d4960 to 0x92e4663, (0xfd03 bytes) 2024-04-23T21:22:26.685 INFO:teuthology.orchestra.run.smithi111.stdout:5137 writesame 0x9240ff8 thru 0x924fdaf data_size 0x4(0xedb8 bytes) 2024-04-23T21:22:26.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:26.695 INFO:teuthology.orchestra.run.smithi111.stdout:5138 punch from 0x386f821 to 0x38778dd, (0x80bc bytes) 2024-04-23T21:22:26.696 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:26.697+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:26.702 INFO:teuthology.orchestra.run.smithi111.stdout:5139 write 0xa62b559 thru 0xa6323d4 (0x6e7c bytes) 2024-04-23T21:22:26.712 INFO:teuthology.orchestra.run.smithi111.stdout:5141 write 0xa8fdc5e thru 0xa90948b (0xb82e bytes) 2024-04-23T21:22:26.716 INFO:teuthology.orchestra.run.smithi111.stdout:5142 read 0x3663175 thru 0x366f300 (0xc18c bytes) 2024-04-23T21:22:26.725 INFO:teuthology.orchestra.run.smithi111.stdout:5143 writesame 0xd56d23b thru 0xd577e97 data_size 0x19(0xac5d bytes) 2024-04-23T21:22:26.725 INFO:teuthology.orchestra.run.smithi111.stdout:5145 write 0xae1694f thru 0xae249f4 (0xe0a6 bytes) 2024-04-23T21:22:26.728 INFO:teuthology.orchestra.run.smithi111.stdout:5146 write 0x5f0a78b thru 0x5f0d4b2 (0x2d28 bytes) 2024-04-23T21:22:26.735 INFO:teuthology.orchestra.run.smithi111.stdout:5147 trunc from 0xd577e98 to 0xa35fce7 2024-04-23T21:22:26.746 INFO:teuthology.orchestra.run.smithi111.stdout:5148 read 0x7654da4 thru 0x765ae60 (0x60bd bytes) 2024-04-23T21:22:26.754 INFO:teuthology.orchestra.run.smithi111.stdout:5149 trunc from 0xa35fce7 to 0x37a503e 2024-04-23T21:22:26.777 INFO:teuthology.orchestra.run.smithi111.stdout:5150 trunc from 0x37a503e to 0x1c58e0d 2024-04-23T21:22:26.797 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:26.798+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:26.803 INFO:teuthology.orchestra.run.smithi111.stdout:5151 trunc from 0x1c58e0d to 0x11a910e 2024-04-23T21:22:26.816 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.816 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.829 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:26.842 INFO:teuthology.orchestra.run.smithi111.stdout:5152 read 0xff90a0 thru 0xffde0b (0x4d6c bytes) 2024-04-23T21:22:26.845 INFO:teuthology.orchestra.run.smithi111.stdout:5153 trunc from 0x11a910e to 0x61842b9 2024-04-23T21:22:26.851 INFO:teuthology.orchestra.run.smithi111.stdout:5155 write 0x2a54149 thru 0x2a56048 (0x1f00 bytes) 2024-04-23T21:22:26.854 INFO:teuthology.orchestra.run.smithi111.stdout:5156 punch from 0x637487 to 0x6471ba, (0xfd33 bytes) 2024-04-23T21:22:26.859 INFO:teuthology.orchestra.run.smithi111.stdout:5157 trunc from 0x61842b9 to 0x9f7dfe3 2024-04-23T21:22:26.864 INFO:teuthology.orchestra.run.smithi111.stdout:5160 punch from 0x8aa970f to 0x8aad050, (0x3941 bytes) 2024-04-23T21:22:26.864 INFO:teuthology.orchestra.run.smithi111.stdout:5162 writesame 0x7569e58 thru 0x756ae2b data_size 0x2(0xfd4 bytes) 2024-04-23T21:22:26.867 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:26.868 INFO:teuthology.orchestra.run.smithi111.stdout:5164 clone 45 order 20 su 524288 sc 9 2024-04-23T21:22:26.899 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:26.900+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:26.947 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:26.947 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:26.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:27.041 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:27.041+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:27.100 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:27.100 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:27.100 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:27.101+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:27.108 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:27.200 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:27.201+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:27.252 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:27.253 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:27.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:27.302 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:27.303+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:27.329 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T21:22:27.346 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-23T21:22:27.347 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:27.390 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:27.414 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:27.415 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:27.415 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:27.415+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:27.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:27.445 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone44 from 0x9f7dfe3 (overlap 0x11a910e) to 0x54c51d 2024-04-23T21:22:27.503 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:27.504+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:27.562 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:27.562 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:27.581 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:27.581 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:27.581 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:27.581 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:27.591 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:27.604 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:27.605+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:27.615 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:27.615 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:27.639 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #43, image image_client.0-clone43 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent44 2024-04-23T21:22:27.650 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:27.651 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:27.668 INFO:teuthology.orchestra.run.smithi111.stdout:5165 read 0xe6c31a thru 0xe6fba7 (0x388e bytes) 2024-04-23T21:22:27.679 INFO:teuthology.orchestra.run.smithi111.stdout:5166 write 0xa20cd78 thru 0xa2112d6 (0x455f bytes) 2024-04-23T21:22:27.683 INFO:teuthology.orchestra.run.smithi111.stdout:5167 trunc from 0xa2112d7 to 0x9537f88 2024-04-23T21:22:27.688 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:27.688 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:27.705 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:27.705 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:27.705 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:27.705 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:27.705 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:27.705+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:27.713 INFO:teuthology.orchestra.run.smithi111.stdout:5168 write 0xba1a7ae thru 0xba1b155 (0x9a8 bytes) 2024-04-23T21:22:27.715 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:5169 read 0x1912063 thru 0x192200d (0xffab bytes) 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:5171 punch from 0x732b650 to 0x7337361, (0xbd11 bytes) 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:5172 read 0x570406c thru 0x57136bd (0xf652 bytes) 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:5173 punch from 0x346659b to 0x3467024, (0xa89 bytes) 2024-04-23T21:22:27.716 INFO:teuthology.orchestra.run.smithi111.stdout:5174 read 0x4a8ade5 thru 0x4a8e57b (0x3797 bytes) 2024-04-23T21:22:27.717 INFO:teuthology.orchestra.run.smithi111.stdout:5175 trunc from 0xba1b156 to 0xce8ea58 2024-04-23T21:22:27.722 INFO:teuthology.orchestra.run.smithi111.stdout:5176 read 0x2969a71 thru 0x2977643 (0xdbd3 bytes) 2024-04-23T21:22:27.722 INFO:teuthology.orchestra.run.smithi111.stdout:5177 write 0x42453e5 thru 0x4253241 (0xde5d bytes) 2024-04-23T21:22:27.723 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:27.724 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:27.726 INFO:teuthology.orchestra.run.smithi111.stdout:5178 writesame 0x985d9e4 thru 0x9866cf4 data_size 0x1(0x9311 bytes) 2024-04-23T21:22:27.744 INFO:teuthology.orchestra.run.smithi111.stdout:5179 read 0x9f50161 thru 0x9f58d50 (0x8bf0 bytes) 2024-04-23T21:22:27.753 INFO:teuthology.orchestra.run.smithi111.stdout:5180 trunc from 0xce8ea58 to 0x63e4448 2024-04-23T21:22:27.761 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:27.761 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:27.777 INFO:teuthology.orchestra.run.smithi111.stdout:5181 writesame 0x8dd79cd thru 0x8de0425 data_size 0x1(0x8a59 bytes) 2024-04-23T21:22:27.777 INFO:teuthology.orchestra.run.smithi111.stdout:5183 write 0x321eddf thru 0x322d373 (0xe595 bytes) 2024-04-23T21:22:27.781 INFO:teuthology.orchestra.run.smithi111.stdout:5184 punch from 0x876c6a0 to 0x876e420, (0x1d80 bytes) 2024-04-23T21:22:27.781 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:27.782 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:27.788 INFO:teuthology.orchestra.run.smithi111.stdout:5185 writesame 0xa48fff8 thru 0xa490275 data_size 0xb(0x27e bytes) 2024-04-23T21:22:27.789 INFO:teuthology.orchestra.run.smithi111.stdout:5186 trunc from 0xa490276 to 0x9f3f41d 2024-04-23T21:22:27.799 INFO:teuthology.orchestra.run.smithi111.stdout:5187 writesame 0xa68a939 thru 0xa69a0af data_size 0x3(0xf777 bytes) 2024-04-23T21:22:27.799 INFO:teuthology.orchestra.run.smithi111.stdout:5189 read 0x62ab3d6 thru 0x62b71f0 (0xbe1b bytes) 2024-04-23T21:22:27.805 INFO:teuthology.orchestra.run.smithi111.stdout:5190 write 0xafc0297 thru 0xafc109a (0xe04 bytes) 2024-04-23T21:22:27.805 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:27.806+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:27.808 INFO:teuthology.orchestra.run.smithi111.stdout:5192 writesame 0x90e30ac thru 0x90e92d7 data_size 0x4(0x622c bytes) 2024-04-23T21:22:27.815 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:27.816 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:27.823 INFO:teuthology.orchestra.run.smithi111.stdout:5194 write 0xebefb32 thru 0xebf403f (0x450e bytes) 2024-04-23T21:22:27.826 INFO:teuthology.orchestra.run.smithi111.stdout:5195 flatten 2024-04-23T21:22:27.830 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-23T21:22:27.606553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2225464108,"num_objects":2025,"num_object_clones":455,"num_object_copies":4050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":954,"num_read":158170,"num_read_kb":900199,"num_write":15597,"num_write_kb":540255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12780,"ondisk_log_size":12780,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":389232,"kb_used_data":89876,"kb_used_omap":17,"kb_used_meta":299246,"kb_avail":754585488,"statfs":{"total":773094113280,"available":772695539712,"internally_reserved":0,"allocated":92033024,"data_stored":848065944,"data_compressed":50305832,"data_compressed_allocated":56365056,"data_compressed_original":819483710,"omap_allocated":17904,"internal_metadata":306428432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":91493601,"num_objects":233,"num_object_clones":100,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":91,"num_read":47246,"num_read_kb":177169,"num_write":1678,"num_write_kb":86185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000599"},"pg_stats":[{"pgid":"3.1f","version":"79'408","reported_seq":3794,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.537156+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:22:23.537156+0000","last_peered":"2024-04-23T21:22:23.537156+0000","last_clean":"2024-04-23T21:22:23.537156+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:23.537156+0000","last_undegraded":"2024-04-23T21:22:23.537156+0000","last_fullsized":"2024-04-23T21:22:23.537156+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:41:05.759971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":72283632,"num_objects":53,"num_object_clones":12,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":30,"num_read":5196,"num_read_kb":15156,"num_write":473,"num_write_kb":33020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"78'347","reported_seq":6273,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619907+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:22:23.619907+0000","last_peered":"2024-04-23T21:22:23.619907+0000","last_clean":"2024-04-23T21:22:23.619907+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:22:23.619907+0000","last_undegraded":"2024-04-23T21:22:23.619907+0000","last_fullsized":"2024-04-23T21:22:23.619907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:58.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":76036159,"num_objects":69,"num_object_clones":18,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":31,"num_read":6314,"num_read_kb":46695,"num_write":407,"num_write_kb":22746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"79'267","reported_seq":7782,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.618455+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:22:23.618455+0000","last_peered":"2024-04-23T21:22:23.618455+0000","last_clean":"2024-04-23T21:22:23.618455+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:23.618455+0000","last_undegraded":"2024-04-23T21:22:23.618455+0000","last_fullsized":"2024-04-23T21:22:23.618455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:00:15.585927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":45837742,"num_objects":64,"num_object_clones":14,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":32,"num_read":7953,"num_read_kb":30246,"num_write":251,"num_write_kb":7623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"79'363","reported_seq":2613,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.538939+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:22:23.538939+0000","last_peered":"2024-04-23T21:22:23.538939+0000","last_clean":"2024-04-23T21:22:23.538939+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:23.538939+0000","last_undegraded":"2024-04-23T21:22:23.538939+0000","last_fullsized":"2024-04-23T21:22:23.538939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:08.160454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":68739163,"num_objects":77,"num_object_clones":20,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":38,"num_read":2626,"num_read_kb":23779,"num_write":430,"num_write_kb":16564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"79'302","reported_seq":9457,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.621096+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:22:23.621096+0000","last_peered":"2024-04-23T21:22:23.621096+0000","last_clean":"2024-04-23T21:22:23.621096+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:23.621096+0000","last_undegraded":"2024-04-23T21:22:23.621096+0000","last_fullsized":"2024-04-23T21:22:23.621096+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:26.274704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":102902046,"num_objects":72,"num_object_clones":21,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":30,"num_read":10176,"num_read_kb":49109,"num_write":313,"num_write_kb":55674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"79'688","reported_seq":4475,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.530962+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:22:23.530962+0000","last_peered":"2024-04-23T21:22:23.530962+0000","last_clean":"2024-04-23T21:22:23.530962+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:23.530962+0000","last_undegraded":"2024-04-23T21:22:23.530962+0000","last_fullsized":"2024-04-23T21:22:23.530962+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:46:51.545637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":51160656,"num_objects":58,"num_object_clones":11,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":28,"num_read":7715,"num_read_kb":38228,"num_write":876,"num_write_kb":26266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"78'474","reported_seq":1871,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.687407+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:22:22.687407+0000","last_peered":"2024-04-23T21:22:22.687407+0000","last_clean":"2024-04-23T21:22:22.687407+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:22:22.687407+0000","last_undegraded":"2024-04-23T21:22:22.687407+0000","last_fullsized":"2024-04-23T21:22:22.687407+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:01:02.892336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":44822586,"num_objects":63,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":34,"num_read":3234,"num_read_kb":12959,"num_write":627,"num_write_kb":9141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"79'607","reported_seq":5269,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619001+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:22:23.619001+0000","last_peered":"2024-04-23T21:22:23.619001+0000","last_clean":"2024-04-23T21:22:23.619001+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:22:23.619001+0000","last_undegraded":"2024-04-23T21:22:23.619001+0000","last_fullsized":"2024-04-23T21:22:23.619001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:37:55.350505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":43096708,"num_objects":68,"num_object_clones":17,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":35,"num_read":7607,"num_read_kb":23548,"num_write":789,"num_write_kb":6270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"79'414","reported_seq":2297,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.426231+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:22:23.426231+0000","last_peered":"2024-04-23T21:22:23.426231+0000","last_clean":"2024-04-23T21:22:23.426231+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:23.426231+0000","last_undegraded":"2024-04-23T21:22:23.426231+0000","last_fullsized":"2024-04-23T21:22:23.426231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:45:10.727013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":44396531,"num_objects":67,"num_object_clones":11,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":35,"num_read":3449,"num_read_kb":30339,"num_write":469,"num_write_kb":27439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"78'259","reported_seq":3665,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.618957+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:22:23.618957+0000","last_peered":"2024-04-23T21:22:23.618957+0000","last_clean":"2024-04-23T21:22:23.618957+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:23.618957+0000","last_undegraded":"2024-04-23T21:22:23.618957+0000","last_fullsized":"2024-04-23T21:22:23.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:51:51.815704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":40704080,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":30,"num_read":3554,"num_read_kb":21740,"num_write":285,"num_write_kb":6022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"78'312","reported_seq":3739,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313299+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:22:23.313299+0000","last_peered":"2024-04-23T21:22:23.313299+0000","last_clean":"2024-04-23T21:22:23.313299+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:22:23.313299+0000","last_undegraded":"2024-04-23T21:22:23.313299+0000","last_fullsized":"2024-04-23T21:22:23.313299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:38.403295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":65559929,"num_objects":69,"num_object_clones":18,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":32,"num_read":3925,"num_read_kb":14174,"num_write":303,"num_write_kb":8412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"79'504","reported_seq":4756,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619659+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:22:23.619659+0000","last_peered":"2024-04-23T21:22:23.619659+0000","last_clean":"2024-04-23T21:22:23.619659+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:23.619659+0000","last_undegraded":"2024-04-23T21:22:23.619659+0000","last_fullsized":"2024-04-23T21:22:23.619659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:05.403863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":105006093,"num_objects":60,"num_object_clones":11,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":29,"num_read":5896,"num_read_kb":40969,"num_write":644,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300078+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:22:23.300078+0000","last_peered":"2024-04-23T21:22:23.300078+0000","last_clean":"2024-04-23T21:22:23.300078+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:22:23.300078+0000","last_undegraded":"2024-04-23T21:22:23.300078+0000","last_fullsized":"2024-04-23T21:22:23.300078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"78'374","reported_seq":2307,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.325179+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:22:23.325179+0000","last_peered":"2024-04-23T21:22:23.325179+0000","last_clean":"2024-04-23T21:22:23.325179+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:22:23.325179+0000","last_undegraded":"2024-04-23T21:22:23.325179+0000","last_fullsized":"2024-04-23T21:22:23.325179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:28:27.828481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":61002405,"num_objects":62,"num_object_clones":15,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":25,"num_read":2549,"num_read_kb":17923,"num_write":453,"num_write_kb":10898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313180+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:23.313180+0000","last_peered":"2024-04-23T21:22:23.313180+0000","last_clean":"2024-04-23T21:22:23.313180+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:23.313180+0000","last_undegraded":"2024-04-23T21:22:23.313180+0000","last_fullsized":"2024-04-23T21:22:23.313180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"78'274","reported_seq":3098,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.495587+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:22.495587+0000","last_peered":"2024-04-23T21:22:22.495587+0000","last_clean":"2024-04-23T21:22:22.495587+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:22.495587+0000","last_undegraded":"2024-04-23T21:22:22.495587+0000","last_fullsized":"2024-04-23T21:22:22.495587+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:13:48.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":84736790,"num_objects":57,"num_object_clones":16,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":24,"num_read":3182,"num_read_kb":30334,"num_write":241,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300965+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:23.300965+0000","last_peered":"2024-04-23T21:22:23.300965+0000","last_clean":"2024-04-23T21:22:23.300965+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:23.300965+0000","last_undegraded":"2024-04-23T21:22:23.300965+0000","last_fullsized":"2024-04-23T21:22:23.300965+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"79'327","reported_seq":2863,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.456449+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:22:23.456449+0000","last_peered":"2024-04-23T21:22:23.456449+0000","last_clean":"2024-04-23T21:22:23.456449+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:22:23.456449+0000","last_undegraded":"2024-04-23T21:22:23.456449+0000","last_fullsized":"2024-04-23T21:22:23.456449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:58:46.312782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":55399920,"num_objects":60,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":30,"num_read":3028,"num_read_kb":13418,"num_write":403,"num_write_kb":6682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379823+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:22:23.379823+0000","last_peered":"2024-04-23T21:22:23.379823+0000","last_clean":"2024-04-23T21:22:23.379823+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:22:23.379823+0000","last_undegraded":"2024-04-23T21:22:23.379823+0000","last_fullsized":"2024-04-23T21:22:23.379823+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"78'286","reported_seq":3686,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.842955+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:22.842955+0000","last_peered":"2024-04-23T21:22:22.842955+0000","last_clean":"2024-04-23T21:22:22.842955+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:22.842955+0000","last_undegraded":"2024-04-23T21:22:22.842955+0000","last_fullsized":"2024-04-23T21:22:22.842955+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:28.348453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":89410994,"num_objects":77,"num_object_clones":18,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":39,"num_read":5650,"num_read_kb":29226,"num_write":305,"num_write_kb":17124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"79'341","reported_seq":7727,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.620944+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:22:23.620944+0000","last_peered":"2024-04-23T21:22:23.620944+0000","last_clean":"2024-04-23T21:22:23.620944+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:23.620944+0000","last_undegraded":"2024-04-23T21:22:23.620944+0000","last_fullsized":"2024-04-23T21:22:23.620944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:00:54.546039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":149341517,"num_objects":79,"num_object_clones":20,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":32,"num_read":7747,"num_read_kb":73553,"num_write":451,"num_write_kb":35543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":293,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313087+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:23.313087+0000","last_peered":"2024-04-23T21:22:23.313087+0000","last_clean":"2024-04-23T21:22:23.313087+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:23.313087+0000","last_undegraded":"2024-04-23T21:22:23.313087+0000","last_fullsized":"2024-04-23T21:22:23.313087+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.292585+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:22.292585+0000","last_peered":"2024-04-23T21:22:22.292585+0000","last_clean":"2024-04-23T21:22:22.292585+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:22.292585+0000","last_undegraded":"2024-04-23T21:22:22.292585+0000","last_fullsized":"2024-04-23T21:22:22.292585+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"78'263","reported_seq":806,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.827761+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:22.827761+0000","last_peered":"2024-04-23T21:22:22.827761+0000","last_clean":"2024-04-23T21:22:22.827761+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:22.827761+0000","last_undegraded":"2024-04-23T21:22:22.827761+0000","last_fullsized":"2024-04-23T21:22:22.827761+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:00:10.118780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":27978992,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":25,"num_read":3285,"num_read_kb":9453,"num_write":307,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313150+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:23.313150+0000","last_peered":"2024-04-23T21:22:23.313150+0000","last_clean":"2024-04-23T21:22:23.313150+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:23.313150+0000","last_undegraded":"2024-04-23T21:22:23.313150+0000","last_fullsized":"2024-04-23T21:22:23.313150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"79'607","reported_seq":7437,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.622313+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:22:23.622313+0000","last_peered":"2024-04-23T21:22:23.622313+0000","last_clean":"2024-04-23T21:22:23.622313+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:23.622313+0000","last_undegraded":"2024-04-23T21:22:23.622313+0000","last_fullsized":"2024-04-23T21:22:23.622313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:37:01.200688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":87760278,"num_objects":78,"num_object_clones":21,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":32,"num_read":9736,"num_read_kb":56381,"num_write":868,"num_write_kb":32782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.324875+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:23.324875+0000","last_peered":"2024-04-23T21:22:23.324875+0000","last_clean":"2024-04-23T21:22:23.324875+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:23.324875+0000","last_undegraded":"2024-04-23T21:22:23.324875+0000","last_fullsized":"2024-04-23T21:22:23.324875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"79'472","reported_seq":4378,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.551391+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:22:23.551391+0000","last_peered":"2024-04-23T21:22:23.551391+0000","last_clean":"2024-04-23T21:22:23.551391+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:22:23.551391+0000","last_undegraded":"2024-04-23T21:22:23.551391+0000","last_fullsized":"2024-04-23T21:22:23.551391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:19.379323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":55607103,"num_objects":47,"num_object_clones":8,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":22,"num_read":6797,"num_read_kb":14151,"num_write":623,"num_write_kb":7939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313087+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:23.313087+0000","last_peered":"2024-04-23T21:22:23.313087+0000","last_clean":"2024-04-23T21:22:23.313087+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:23.313087+0000","last_undegraded":"2024-04-23T21:22:23.313087+0000","last_fullsized":"2024-04-23T21:22:23.313087+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"77'203","reported_seq":2016,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379900+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:22:23.379900+0000","last_peered":"2024-04-23T21:22:23.379900+0000","last_clean":"2024-04-23T21:22:23.379900+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:22:23.379900+0000","last_undegraded":"2024-04-23T21:22:23.379900+0000","last_fullsized":"2024-04-23T21:22:23.379900+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:50:55.721596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":30618672,"num_objects":64,"num_object_clones":18,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":33,"num_read":1933,"num_read_kb":14039,"num_write":219,"num_write_kb":10969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"78'260","reported_seq":6916,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.621270+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:22:23.621270+0000","last_peered":"2024-04-23T21:22:23.621270+0000","last_clean":"2024-04-23T21:22:23.621270+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:23.621270+0000","last_undegraded":"2024-04-23T21:22:23.621270+0000","last_fullsized":"2024-04-23T21:22:23.621270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:11:20.878508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":103445929,"num_objects":72,"num_object_clones":20,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":32,"num_read":6698,"num_read_kb":45882,"num_write":286,"num_write_kb":15594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"79'421","reported_seq":3283,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.411935+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:22:23.411935+0000","last_peered":"2024-04-23T21:22:23.411935+0000","last_clean":"2024-04-23T21:22:23.411935+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:23.411935+0000","last_undegraded":"2024-04-23T21:22:23.411935+0000","last_fullsized":"2024-04-23T21:22:23.411935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:21.509463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":40065512,"num_objects":58,"num_object_clones":10,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":28,"num_read":5262,"num_read_kb":12426,"num_write":422,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'365","reported_seq":2185,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379776+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:22:23.379776+0000","last_peered":"2024-04-23T21:22:23.379776+0000","last_clean":"2024-04-23T21:22:23.379776+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:22:23.379776+0000","last_undegraded":"2024-04-23T21:22:23.379776+0000","last_fullsized":"2024-04-23T21:22:23.379776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:51.717415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":61296972,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":26,"num_read":2554,"num_read_kb":25780,"num_write":404,"num_write_kb":12062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"79'280","reported_seq":1009,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.434041+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:22:23.434041+0000","last_peered":"2024-04-23T21:22:23.434041+0000","last_clean":"2024-04-23T21:22:23.434041+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:22:23.434041+0000","last_undegraded":"2024-04-23T21:22:23.434041+0000","last_fullsized":"2024-04-23T21:22:23.434041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:59.575169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":80032566,"num_objects":60,"num_object_clones":16,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":31,"num_read":1114,"num_read_kb":14927,"num_write":259,"num_write_kb":15541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"78'223","reported_seq":2447,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313316+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:22:23.313316+0000","last_peered":"2024-04-23T21:22:23.313316+0000","last_clean":"2024-04-23T21:22:23.313316+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:23.313316+0000","last_undegraded":"2024-04-23T21:22:23.313316+0000","last_fullsized":"2024-04-23T21:22:23.313316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:21:06.090657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":42361659,"num_objects":51,"num_object_clones":9,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":28,"num_read":2453,"num_read_kb":11389,"num_write":201,"num_write_kb":2716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"78'306","reported_seq":1208,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313479+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:22:23.313479+0000","last_peered":"2024-04-23T21:22:23.313479+0000","last_clean":"2024-04-23T21:22:23.313479+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:23.313479+0000","last_undegraded":"2024-04-23T21:22:23.313479+0000","last_fullsized":"2024-04-23T21:22:23.313479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:26:49.126009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":36674923,"num_objects":44,"num_object_clones":7,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":23,"num_read":1797,"num_read_kb":8411,"num_write":357,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"79'854","reported_seq":9620,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.620002+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:22:23.620002+0000","last_peered":"2024-04-23T21:22:23.620002+0000","last_clean":"2024-04-23T21:22:23.620002+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:23.620002+0000","last_undegraded":"2024-04-23T21:22:23.620002+0000","last_fullsized":"2024-04-23T21:22:23.620002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:26:22.317940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":118035912,"num_objects":75,"num_object_clones":16,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":30,"num_read":12882,"num_read_kb":57742,"num_write":1235,"num_write_kb":15540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"79'359","reported_seq":2571,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300337+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:22:23.300337+0000","last_peered":"2024-04-23T21:22:23.300337+0000","last_clean":"2024-04-23T21:22:23.300337+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:23.300337+0000","last_undegraded":"2024-04-23T21:22:23.300337+0000","last_fullsized":"2024-04-23T21:22:23.300337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:15.735525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":97089947,"num_objects":68,"num_object_clones":16,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":30,"num_read":3001,"num_read_kb":23044,"num_write":430,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"79'233","reported_seq":2069,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.437389+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:22:23.437389+0000","last_peered":"2024-04-23T21:22:23.437389+0000","last_clean":"2024-04-23T21:22:23.437389+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:23.437389+0000","last_undegraded":"2024-04-23T21:22:23.437389+0000","last_fullsized":"2024-04-23T21:22:23.437389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:25.000275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":87772664,"num_objects":60,"num_object_clones":15,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":25,"num_read":1790,"num_read_kb":20042,"num_write":268,"num_write_kb":14124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"79'473","reported_seq":2343,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.415895+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:22:23.415895+0000","last_peered":"2024-04-23T21:22:23.415895+0000","last_clean":"2024-04-23T21:22:23.415895+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:23.415895+0000","last_undegraded":"2024-04-23T21:22:23.415895+0000","last_fullsized":"2024-04-23T21:22:23.415895+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:59:39.713723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":72590514,"num_objects":66,"num_object_clones":10,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":26,"num_read":3829,"num_read_kb":36394,"num_write":610,"num_write_kb":19593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"79'797","reported_seq":2637,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.552700+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:22:23.552700+0000","last_peered":"2024-04-23T21:22:23.552700+0000","last_clean":"2024-04-23T21:22:23.552700+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:23.552700+0000","last_undegraded":"2024-04-23T21:22:23.552700+0000","last_fullsized":"2024-04-23T21:22:23.552700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:19.290204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":83105146,"num_objects":70,"num_object_clones":14,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":5122,"num_read_kb":38642,"num_write":1229,"num_write_kb":23791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2224873740,"num_objects":2023,"num_object_clones":455,"num_object_copies":4046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":954,"num_read":158054,"num_read_kb":900099,"num_write":15438,"num_write_kb":537839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84684800,"data_stored":843464538,"data_compressed":50229476,"data_compressed_allocated":56201216,"data_compressed_original":818171312,"omap_allocated":0,"internal_metadata":0},"log_size":12663,"ondisk_log_size":12663,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58284,"kb_used_data":13220,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94313556,"statfs":{"total":96636764160,"available":96577081344,"internally_reserved":0,"allocated":13537280,"data_stored":109891485,"data_compressed":7373241,"data_compressed_allocated":8323072,"data_compressed_original":105761647,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35736,"kb_used_data":4552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336104,"statfs":{"total":96636764160,"available":96600170496,"internally_reserved":0,"allocated":4661248,"data_stored":39866008,"data_compressed":2093626,"data_compressed_allocated":2482176,"data_compressed_original":38250387,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48660,"kb_used_data":9152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323180,"statfs":{"total":96636764160,"available":96586936320,"internally_reserved":0,"allocated":9371648,"data_stored":82139560,"data_compressed":4627598,"data_compressed_allocated":5382144,"data_compressed_original":79023297,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70912,"kb_used_data":21688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94300928,"statfs":{"total":96636764160,"available":96564150272,"internally_reserved":0,"allocated":22208512,"data_stored":233826812,"data_compressed":13755609,"data_compressed_allocated":14831616,"data_compressed_original":227589506,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54612,"kb_used_data":10952,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94317228,"statfs":{"total":96636764160,"available":96580841472,"internally_reserved":0,"allocated":11214848,"data_stored":88298549,"data_compressed":4967631,"data_compressed_allocated":5799936,"data_compressed_original":83783582,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36044,"kb_used_data":4864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335796,"statfs":{"total":96636764160,"available":96599855104,"internally_reserved":0,"allocated":4980736,"data_stored":36036521,"data_compressed":2125865,"data_compressed_allocated":2441216,"data_compressed_original":34212278,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":8332,"kb_used_omap":6,"kb_used_meta":32505,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":8531968,"data_stored":87139627,"data_compressed":5294755,"data_compressed_allocated":5435392,"data_compressed_original":84836026,"omap_allocated":6748,"internal_metadata":33285540},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44132,"kb_used_data":17116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327708,"statfs":{"total":96636764160,"available":96591572992,"internally_reserved":0,"allocated":17526784,"data_stored":170867382,"data_compressed":10067507,"data_compressed_allocated":11669504,"data_compressed_original":166026987,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16613376,"data_stored":170539900,"data_compressed":10076774,"data_compressed_allocated":11669504,"data_compressed_original":166125291,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7561216,"data_stored":85500553,"data_compressed":5196367,"data_compressed_allocated":5316608,"data_compressed_original":83566266,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":35604935,"data_compressed":2124504,"data_compressed_allocated":2437120,"data_compressed_original":34183606,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10301440,"data_stored":88010323,"data_compressed":4984973,"data_compressed_allocated":5799936,"data_compressed_original":83921168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21295104,"data_stored":233499325,"data_compressed":13764886,"data_compressed_allocated":14831616,"data_compressed_original":227687810,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":81877572,"data_compressed":4643053,"data_compressed_allocated":5382144,"data_compressed_original":79187137,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3743744,"data_stored":39440180,"data_compressed":2093626,"data_compressed_allocated":2482176,"data_compressed_original":38250387,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12632064,"data_stored":108991750,"data_compressed":7345293,"data_compressed_allocated":8282112,"data_compressed_original":105249647,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:27.832 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:27.842 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:27.843 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:27.843 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:27.843 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:27.843 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:27.853 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:27.855 INFO:teuthology.orchestra.run.smithi111.stdout:5196 writesame 0xb08a0de thru 0xb090f8f data_size 0x3(0x6eb2 bytes) 2024-04-23T21:22:27.862 INFO:teuthology.orchestra.run.smithi111.stdout:5197 read 0xc5ae7ad thru 0xc5b7868 (0x90bc bytes) 2024-04-23T21:22:27.862 INFO:teuthology.orchestra.run.smithi111.stdout:5200 punch from 0xb4f0d5f to 0xb4fa8dc, (0x9b7d bytes) 2024-04-23T21:22:27.862 INFO:teuthology.orchestra.run.smithi111.stdout:5201 write 0x4cf71d thru 0x4d8098 (0x897c bytes) 2024-04-23T21:22:27.863 INFO:teuthology.orchestra.run.smithi111.stdout:5203 read 0xdad79e3 thru 0xdae27f6 (0xae14 bytes) 2024-04-23T21:22:27.872 INFO:teuthology.orchestra.run.smithi111.stdout:5204 trunc from 0xebf4040 to 0x775af35 2024-04-23T21:22:27.882 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:27.896 INFO:teuthology.orchestra.run.smithi111.stdout:5207 write 0xd7c6793 thru 0xd7cdbef (0x745d bytes) 2024-04-23T21:22:27.899 INFO:teuthology.orchestra.run.smithi111.stdout:5208 punch from 0xcb5e661 to 0xcb64829, (0x61c8 bytes) 2024-04-23T21:22:27.899 INFO:teuthology.orchestra.run.smithi111.stdout:5209 read 0x5c51730 thru 0x5c5b51c (0x9ded bytes) 2024-04-23T21:22:27.905 INFO:teuthology.orchestra.run.smithi111.stdout:5210 writesame 0xdfe27d3 thru 0xdfe7e28 data_size 0x2(0x5656 bytes) 2024-04-23T21:22:27.905 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:27.905 INFO:teuthology.orchestra.run.smithi111.stdout:5213 write 0x1858b03 thru 0x1864a6e (0xbf6c bytes) 2024-04-23T21:22:27.906 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:27.906+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:27.914 INFO:teuthology.orchestra.run.smithi111.stdout:5214 writesame 0xe95d00a thru 0xe9652f7 data_size 0x1(0x82ee bytes) 2024-04-23T21:22:27.914 INFO:teuthology.orchestra.run.smithi111.stdout:5215 write 0x3bbb78 thru 0x3c8e5d (0xd2e6 bytes) 2024-04-23T21:22:27.917 INFO:teuthology.orchestra.run.smithi111.stdout:5217 read 0x3aa48da thru 0x3aaa4c5 (0x5bec bytes) 2024-04-23T21:22:27.933 INFO:teuthology.orchestra.run.smithi111.stdout:5219 trunc from 0xe9652f8 to 0x1c86bdb 2024-04-23T21:22:27.966 INFO:teuthology.orchestra.run.smithi111.stdout:5220 writesame 0xdef3ff1 thru 0xdefc7e2 data_size 0x1(0x87f2 bytes) 2024-04-23T21:22:27.966 INFO:teuthology.orchestra.run.smithi111.stdout:5221 write 0x7545fce thru 0x7550aea (0xab1d bytes) 2024-04-23T21:22:27.972 INFO:teuthology.orchestra.run.smithi111.stdout:5222 trunc from 0xdefc7e3 to 0xbc04a47 2024-04-23T21:22:27.973 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:27.973 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:27.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:27.973 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.333690152999999, 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.974 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:27.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.333658045999996, 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.978 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:27.982 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.333634660000001, 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:27.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.986 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.333614353000002, 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.987 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:27.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:27.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:27.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.333594540999997, 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.991 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332523143000003, 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.995 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332489797999997, 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:27.999 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:28.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332458269, 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:28.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:28.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:28.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332430879999997, 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332418973999999, 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.011 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:28.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:28.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:28.015 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332400730000003, 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:28.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332390912999998, 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:28.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:28.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:28.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332380665999999, 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.024 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332371015, 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.028 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:28.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.332337516000003, 2024-04-23T21:22:28.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:28.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.328712092000004, 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:28.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:28.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.328694222999999, 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.041 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:28.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:28.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.328675494000002, 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.045 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:28.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.328657065999998, 2024-04-23T21:22:28.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.053 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.328639160999998, 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:28.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:28.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.057 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.058 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:28.058 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:28.058 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5225 punch from 0x73d999e to 0x73e1329, (0x798b bytes) 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5226 read 0x5a05bde thru 0x5a11996 (0xbdb9 bytes) 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5227 write 0xa90218c thru 0xa90768b (0x5500 bytes) 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5228 punch from 0xb0c0be9 to 0xb0c1863, (0xc7a bytes) 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5229 read 0x7cd9582 thru 0x7cde20b (0x4c8a bytes) 2024-04-23T21:22:28.059 INFO:teuthology.orchestra.run.smithi111.stdout:5232 write 0xa73f07a thru 0xa74a59b (0xb522 bytes) 2024-04-23T21:22:28.060 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:28.062 INFO:teuthology.orchestra.run.smithi111.stdout:5234 punch from 0x3cb0e32 to 0x3cb5257, (0x4425 bytes) 2024-04-23T21:22:28.062 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5236 punch from 0x40b2f68 to 0x40bd7f7, (0xa88f bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5237 read 0xb59e955 thru 0xb5ad6b5 (0xed61 bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5240 read 0x43ab1f0 thru 0x43b20c6 (0x6ed7 bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5243 write 0xee5f482 thru 0xee62b40 (0x36bf bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5249 read 0x3e5499e thru 0x3e6273c (0xdd9f bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5251 writesame 0x1f99250 thru 0x1fa683a data_size 0x1(0xd5eb bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5252 writesame 0x25f1b9 thru 0x26892b data_size 0x1(0x9773 bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5253 read 0x7ac1121 thru 0x7ad108d (0xff6d bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5254 read 0x1c30f6a thru 0x1c398f1 (0x8988 bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5255 write 0xe3140c4 thru 0xe317c7f (0x3bbc bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5257 write 0x62b12a6 thru 0x62b9007 (0x7d62 bytes) 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.063 INFO:teuthology.orchestra.run.smithi111.stdout:5261 write 0x80cd9c0 thru 0x80d3708 (0x5d49 bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5262 punch from 0x34d5621 to 0x34d8a1a, (0x33f9 bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5263 read 0xe4a1966 thru 0xe4aad43 (0x93de bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5265 write 0xe5ac3e6 thru 0xe5b95b6 (0xd1d1 bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5266 punch from 0x569e81e to 0x569f8ee, (0x10d0 bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5267 read 0x6f6d663 thru 0x6f74747 (0x70e5 bytes) 2024-04-23T21:22:28.064 INFO:teuthology.orchestra.run.smithi111.stdout:5268 read 0xa66682e thru 0xa673bdc (0xd3af bytes) 2024-04-23T21:22:28.064 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:28.059+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:28.074 INFO:teuthology.orchestra.run.smithi111.stdout:5272 trunc from 0xee62b41 to 0x1174380 2024-04-23T21:22:28.084 INFO:teuthology.orchestra.run.smithi111.stdout:5273 punch from 0x8e6409 to 0x8f48c6, (0xe4bd bytes) 2024-04-23T21:22:28.084 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.093 INFO:teuthology.orchestra.run.smithi111.stdout:5275 write 0x342681e thru 0x34293dc (0x2bbf bytes) 2024-04-23T21:22:28.096 INFO:teuthology.orchestra.run.smithi111.stdout:5276 write 0x33e57ee thru 0x33e5c88 (0x49b bytes) 2024-04-23T21:22:28.099 INFO:teuthology.orchestra.run.smithi111.stdout:5278 read 0x21891cf thru 0x2189ffa (0xe2c bytes) 2024-04-23T21:22:28.119 INFO:teuthology.orchestra.run.smithi111.stdout:5279 write 0xee5b901 thru 0xee629e3 (0x70e3 bytes) 2024-04-23T21:22:28.123 INFO:teuthology.orchestra.run.smithi111.stdout:5280 write 0xbb7a251 thru 0xbb86e52 (0xcc02 bytes) 2024-04-23T21:22:28.129 INFO:teuthology.orchestra.run.smithi111.stdout:5281 write 0x10225b3 thru 0x1023025 (0xa73 bytes) 2024-04-23T21:22:28.132 INFO:teuthology.orchestra.run.smithi111.stdout:5282 writesame 0x55ed39b thru 0x55ef1b7 data_size 0x1(0x1e1d bytes) 2024-04-23T21:22:28.137 INFO:teuthology.orchestra.run.smithi111.stdout:5284 punch from 0x3c1d44f to 0x3c21dec, (0x499d bytes) 2024-04-23T21:22:28.137 INFO:teuthology.orchestra.run.smithi111.stdout:5285 read 0xdc04db7 thru 0xdc083a4 (0x35ee bytes) 2024-04-23T21:22:28.138 INFO:teuthology.orchestra.run.smithi111.stdout:5287 writesame 0xe2d1202 thru 0xe2de889 data_size 0x14(0xd688 bytes) 2024-04-23T21:22:28.141 INFO:teuthology.orchestra.run.smithi111.stdout:5288 write 0xce524a5 thru 0xce5a108 (0x7c64 bytes) 2024-04-23T21:22:28.145 INFO:teuthology.orchestra.run.smithi111.stdout:5289 trunc from 0xee629e4 to 0xeade3b0 2024-04-23T21:22:28.155 INFO:teuthology.orchestra.run.smithi111.stdout:5290 writesame 0xbfc4c63 thru 0xbfc973f data_size 0x1(0x4add bytes) 2024-04-23T21:22:28.159 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:28.159 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:28.161 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:28.162+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:28.166 INFO:teuthology.orchestra.run.smithi111.stdout:5291 read 0x64bd9a7 thru 0x64bdb9e (0x1f8 bytes) 2024-04-23T21:22:28.166 INFO:teuthology.orchestra.run.smithi111.stdout:5293 write 0x21826c thru 0x2279c2 (0xf757 bytes) 2024-04-23T21:22:28.168 INFO:teuthology.orchestra.run.smithi111.stdout:5294 writesame 0xdf073fd thru 0xdf0fb48 data_size 0xe(0x874c bytes) 2024-04-23T21:22:28.171 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-23T21:22:27.606553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2225464108,"num_objects":2025,"num_object_clones":455,"num_object_copies":4050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":954,"num_read":158170,"num_read_kb":900199,"num_write":15597,"num_write_kb":540255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12780,"ondisk_log_size":12780,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":389232,"kb_used_data":89876,"kb_used_omap":17,"kb_used_meta":299246,"kb_avail":754585488,"statfs":{"total":773094113280,"available":772695539712,"internally_reserved":0,"allocated":92033024,"data_stored":848065944,"data_compressed":50305832,"data_compressed_allocated":56365056,"data_compressed_original":819483710,"omap_allocated":17904,"internal_metadata":306428432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":91493601,"num_objects":233,"num_object_clones":100,"num_object_copies":466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":91,"num_read":47246,"num_read_kb":177169,"num_write":1678,"num_write_kb":86185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000599"},"pg_stats":[{"pgid":"3.1f","version":"79'408","reported_seq":3794,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.537156+0000","last_change":"2024-04-23T21:21:32.841149+0000","last_active":"2024-04-23T21:22:23.537156+0000","last_peered":"2024-04-23T21:22:23.537156+0000","last_clean":"2024-04-23T21:22:23.537156+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:23.537156+0000","last_undegraded":"2024-04-23T21:22:23.537156+0000","last_fullsized":"2024-04-23T21:22:23.537156+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:41:05.759971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084456899999999998,"stat_sum":{"num_bytes":72283632,"num_objects":53,"num_object_clones":12,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":30,"num_read":5196,"num_read_kb":15156,"num_write":473,"num_write_kb":33020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"78'347","reported_seq":6273,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619907+0000","last_change":"2024-04-23T21:21:19.892046+0000","last_active":"2024-04-23T21:22:23.619907+0000","last_peered":"2024-04-23T21:22:23.619907+0000","last_clean":"2024-04-23T21:22:23.619907+0000","last_became_active":"2024-04-23T21:21:17.774622+0000","last_became_peered":"2024-04-23T21:21:17.774622+0000","last_unstale":"2024-04-23T21:22:23.619907+0000","last_undegraded":"2024-04-23T21:22:23.619907+0000","last_fullsized":"2024-04-23T21:22:23.619907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:58.545957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077979700000000004,"stat_sum":{"num_bytes":76036159,"num_objects":69,"num_object_clones":18,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":31,"num_read":6314,"num_read_kb":46695,"num_write":407,"num_write_kb":22746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"79'267","reported_seq":7782,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.618455+0000","last_change":"2024-04-23T21:21:19.798052+0000","last_active":"2024-04-23T21:22:23.618455+0000","last_peered":"2024-04-23T21:22:23.618455+0000","last_clean":"2024-04-23T21:22:23.618455+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:23.618455+0000","last_undegraded":"2024-04-23T21:22:23.618455+0000","last_fullsized":"2024-04-23T21:22:23.618455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:00:15.585927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012435650000000001,"stat_sum":{"num_bytes":45837742,"num_objects":64,"num_object_clones":14,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":32,"num_read":7953,"num_read_kb":30246,"num_write":251,"num_write_kb":7623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"79'363","reported_seq":2613,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.538939+0000","last_change":"2024-04-23T21:21:19.797761+0000","last_active":"2024-04-23T21:22:23.538939+0000","last_peered":"2024-04-23T21:22:23.538939+0000","last_clean":"2024-04-23T21:22:23.538939+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:23.538939+0000","last_undegraded":"2024-04-23T21:22:23.538939+0000","last_fullsized":"2024-04-23T21:22:23.538939+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:08.160454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001042219,"stat_sum":{"num_bytes":68739163,"num_objects":77,"num_object_clones":20,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":38,"num_read":2626,"num_read_kb":23779,"num_write":430,"num_write_kb":16564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"79'302","reported_seq":9457,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.621096+0000","last_change":"2024-04-23T21:21:32.845215+0000","last_active":"2024-04-23T21:22:23.621096+0000","last_peered":"2024-04-23T21:22:23.621096+0000","last_clean":"2024-04-23T21:22:23.621096+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:23.621096+0000","last_undegraded":"2024-04-23T21:22:23.621096+0000","last_fullsized":"2024-04-23T21:22:23.621096+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:33:26.274704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088836099999999995,"stat_sum":{"num_bytes":102902046,"num_objects":72,"num_object_clones":21,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":30,"num_read":10176,"num_read_kb":49109,"num_write":313,"num_write_kb":55674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"79'688","reported_seq":4475,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.530962+0000","last_change":"2024-04-23T21:21:52.959327+0000","last_active":"2024-04-23T21:22:23.530962+0000","last_peered":"2024-04-23T21:22:23.530962+0000","last_clean":"2024-04-23T21:22:23.530962+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:23.530962+0000","last_undegraded":"2024-04-23T21:22:23.530962+0000","last_fullsized":"2024-04-23T21:22:23.530962+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:46:51.545637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063854700000000005,"stat_sum":{"num_bytes":51160656,"num_objects":58,"num_object_clones":11,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":28,"num_read":7715,"num_read_kb":38228,"num_write":876,"num_write_kb":26266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"78'474","reported_seq":1871,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.687407+0000","last_change":"2024-04-23T21:21:52.959071+0000","last_active":"2024-04-23T21:22:22.687407+0000","last_peered":"2024-04-23T21:22:22.687407+0000","last_clean":"2024-04-23T21:22:22.687407+0000","last_became_active":"2024-04-23T21:21:52.957627+0000","last_became_peered":"2024-04-23T21:21:52.957627+0000","last_unstale":"2024-04-23T21:22:22.687407+0000","last_undegraded":"2024-04-23T21:22:22.687407+0000","last_fullsized":"2024-04-23T21:22:22.687407+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:01:02.892336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":44822586,"num_objects":63,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":34,"num_read":3234,"num_read_kb":12959,"num_write":627,"num_write_kb":9141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1209752,"num_keys_recovered":11,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"79'607","reported_seq":5269,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619001+0000","last_change":"2024-04-23T21:21:19.891568+0000","last_active":"2024-04-23T21:22:23.619001+0000","last_peered":"2024-04-23T21:22:23.619001+0000","last_clean":"2024-04-23T21:22:23.619001+0000","last_became_active":"2024-04-23T21:21:17.776423+0000","last_became_peered":"2024-04-23T21:21:17.776423+0000","last_unstale":"2024-04-23T21:22:23.619001+0000","last_undegraded":"2024-04-23T21:22:23.619001+0000","last_fullsized":"2024-04-23T21:22:23.619001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:37:55.350505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":43096708,"num_objects":68,"num_object_clones":17,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":35,"num_read":7607,"num_read_kb":23548,"num_write":789,"num_write_kb":6270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"79'414","reported_seq":2297,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.426231+0000","last_change":"2024-04-23T21:21:32.841467+0000","last_active":"2024-04-23T21:22:23.426231+0000","last_peered":"2024-04-23T21:22:23.426231+0000","last_clean":"2024-04-23T21:22:23.426231+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:23.426231+0000","last_undegraded":"2024-04-23T21:22:23.426231+0000","last_fullsized":"2024-04-23T21:22:23.426231+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:45:10.727013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096266899999999998,"stat_sum":{"num_bytes":44396531,"num_objects":67,"num_object_clones":11,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":35,"num_read":3449,"num_read_kb":30339,"num_write":469,"num_write_kb":27439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"78'259","reported_seq":3665,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.618957+0000","last_change":"2024-04-23T21:21:19.798191+0000","last_active":"2024-04-23T21:22:23.618957+0000","last_peered":"2024-04-23T21:22:23.618957+0000","last_clean":"2024-04-23T21:22:23.618957+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:23.618957+0000","last_undegraded":"2024-04-23T21:22:23.618957+0000","last_fullsized":"2024-04-23T21:22:23.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:51:51.815704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013217789999999999,"stat_sum":{"num_bytes":40704080,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":30,"num_read":3554,"num_read_kb":21740,"num_write":285,"num_write_kb":6022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"78'312","reported_seq":3739,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313299+0000","last_change":"2024-04-23T21:21:19.892052+0000","last_active":"2024-04-23T21:22:23.313299+0000","last_peered":"2024-04-23T21:22:23.313299+0000","last_clean":"2024-04-23T21:22:23.313299+0000","last_became_active":"2024-04-23T21:21:17.776777+0000","last_became_peered":"2024-04-23T21:21:17.776777+0000","last_unstale":"2024-04-23T21:22:23.313299+0000","last_undegraded":"2024-04-23T21:22:23.313299+0000","last_fullsized":"2024-04-23T21:22:23.313299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:38.403295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000379978,"stat_sum":{"num_bytes":65559929,"num_objects":69,"num_object_clones":18,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":32,"num_read":3925,"num_read_kb":14174,"num_write":303,"num_write_kb":8412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"79'504","reported_seq":4756,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.619659+0000","last_change":"2024-04-23T21:21:19.788773+0000","last_active":"2024-04-23T21:22:23.619659+0000","last_peered":"2024-04-23T21:22:23.619659+0000","last_clean":"2024-04-23T21:22:23.619659+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:23.619659+0000","last_undegraded":"2024-04-23T21:22:23.619659+0000","last_fullsized":"2024-04-23T21:22:23.619659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:05.403863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048758699999999999,"stat_sum":{"num_bytes":105006093,"num_objects":60,"num_object_clones":11,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":29,"num_read":5896,"num_read_kb":40969,"num_write":644,"num_write_kb":12374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300078+0000","last_change":"2024-04-23T21:21:09.734248+0000","last_active":"2024-04-23T21:22:23.300078+0000","last_peered":"2024-04-23T21:22:23.300078+0000","last_clean":"2024-04-23T21:22:23.300078+0000","last_became_active":"2024-04-23T21:21:09.734003+0000","last_became_peered":"2024-04-23T21:21:09.734003+0000","last_unstale":"2024-04-23T21:22:23.300078+0000","last_undegraded":"2024-04-23T21:22:23.300078+0000","last_fullsized":"2024-04-23T21:22:23.300078+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:05:39.249896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"78'374","reported_seq":2307,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.325179+0000","last_change":"2024-04-23T21:21:19.798390+0000","last_active":"2024-04-23T21:22:23.325179+0000","last_peered":"2024-04-23T21:22:23.325179+0000","last_clean":"2024-04-23T21:22:23.325179+0000","last_became_active":"2024-04-23T21:21:17.782054+0000","last_became_peered":"2024-04-23T21:21:17.782054+0000","last_unstale":"2024-04-23T21:22:23.325179+0000","last_undegraded":"2024-04-23T21:22:23.325179+0000","last_fullsized":"2024-04-23T21:22:23.325179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:28:27.828481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001493681,"stat_sum":{"num_bytes":61002405,"num_objects":62,"num_object_clones":15,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":25,"num_read":2549,"num_read_kb":17923,"num_write":453,"num_write_kb":10898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313180+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:23.313180+0000","last_peered":"2024-04-23T21:22:23.313180+0000","last_clean":"2024-04-23T21:22:23.313180+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:23.313180+0000","last_undegraded":"2024-04-23T21:22:23.313180+0000","last_fullsized":"2024-04-23T21:22:23.313180+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"78'274","reported_seq":3098,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.495587+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:22.495587+0000","last_peered":"2024-04-23T21:22:22.495587+0000","last_clean":"2024-04-23T21:22:22.495587+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:22.495587+0000","last_undegraded":"2024-04-23T21:22:22.495587+0000","last_fullsized":"2024-04-23T21:22:22.495587+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:13:48.742023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":84736790,"num_objects":57,"num_object_clones":16,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":24,"num_read":3182,"num_read_kb":30334,"num_write":241,"num_write_kb":16420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300965+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:23.300965+0000","last_peered":"2024-04-23T21:22:23.300965+0000","last_clean":"2024-04-23T21:22:23.300965+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:23.300965+0000","last_undegraded":"2024-04-23T21:22:23.300965+0000","last_fullsized":"2024-04-23T21:22:23.300965+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"79'327","reported_seq":2863,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.456449+0000","last_change":"2024-04-23T21:21:19.891951+0000","last_active":"2024-04-23T21:22:23.456449+0000","last_peered":"2024-04-23T21:22:23.456449+0000","last_clean":"2024-04-23T21:22:23.456449+0000","last_became_active":"2024-04-23T21:21:17.775649+0000","last_became_peered":"2024-04-23T21:21:17.775649+0000","last_unstale":"2024-04-23T21:22:23.456449+0000","last_undegraded":"2024-04-23T21:22:23.456449+0000","last_fullsized":"2024-04-23T21:22:23.456449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:58:46.312782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081057199999999996,"stat_sum":{"num_bytes":55399920,"num_objects":60,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":30,"num_read":3028,"num_read_kb":13418,"num_write":403,"num_write_kb":6682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379823+0000","last_change":"2024-04-23T21:21:09.733971+0000","last_active":"2024-04-23T21:22:23.379823+0000","last_peered":"2024-04-23T21:22:23.379823+0000","last_clean":"2024-04-23T21:22:23.379823+0000","last_became_active":"2024-04-23T21:21:09.733667+0000","last_became_peered":"2024-04-23T21:21:09.733667+0000","last_unstale":"2024-04-23T21:22:23.379823+0000","last_undegraded":"2024-04-23T21:22:23.379823+0000","last_fullsized":"2024-04-23T21:22:23.379823+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:00:41.222768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"78'286","reported_seq":3686,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.842955+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:22.842955+0000","last_peered":"2024-04-23T21:22:22.842955+0000","last_clean":"2024-04-23T21:22:22.842955+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:22.842955+0000","last_undegraded":"2024-04-23T21:22:22.842955+0000","last_fullsized":"2024-04-23T21:22:22.842955+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:13:28.348453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":89410994,"num_objects":77,"num_object_clones":18,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":39,"num_read":5650,"num_read_kb":29226,"num_write":305,"num_write_kb":17124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"79'341","reported_seq":7727,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.620944+0000","last_change":"2024-04-23T21:21:32.844019+0000","last_active":"2024-04-23T21:22:23.620944+0000","last_peered":"2024-04-23T21:22:23.620944+0000","last_clean":"2024-04-23T21:22:23.620944+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:23.620944+0000","last_undegraded":"2024-04-23T21:22:23.620944+0000","last_fullsized":"2024-04-23T21:22:23.620944+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:00:54.546039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000539366,"stat_sum":{"num_bytes":149341517,"num_objects":79,"num_object_clones":20,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":32,"num_read":7747,"num_read_kb":73553,"num_write":451,"num_write_kb":35543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":293,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313087+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:23.313087+0000","last_peered":"2024-04-23T21:22:23.313087+0000","last_clean":"2024-04-23T21:22:23.313087+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:23.313087+0000","last_undegraded":"2024-04-23T21:22:23.313087+0000","last_fullsized":"2024-04-23T21:22:23.313087+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.292585+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:22.292585+0000","last_peered":"2024-04-23T21:22:22.292585+0000","last_clean":"2024-04-23T21:22:22.292585+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:22.292585+0000","last_undegraded":"2024-04-23T21:22:22.292585+0000","last_fullsized":"2024-04-23T21:22:22.292585+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"78'263","reported_seq":806,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-04-23T21:22:22.827761+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:22.827761+0000","last_peered":"2024-04-23T21:22:22.827761+0000","last_clean":"2024-04-23T21:22:22.827761+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:22.827761+0000","last_undegraded":"2024-04-23T21:22:22.827761+0000","last_fullsized":"2024-04-23T21:22:22.827761+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:00:10.118780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":27978992,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":25,"num_read":3285,"num_read_kb":9453,"num_write":307,"num_write_kb":3350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313150+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:23.313150+0000","last_peered":"2024-04-23T21:22:23.313150+0000","last_clean":"2024-04-23T21:22:23.313150+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:23.313150+0000","last_undegraded":"2024-04-23T21:22:23.313150+0000","last_fullsized":"2024-04-23T21:22:23.313150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"79'607","reported_seq":7437,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.622313+0000","last_change":"2024-04-23T21:21:19.788573+0000","last_active":"2024-04-23T21:22:23.622313+0000","last_peered":"2024-04-23T21:22:23.622313+0000","last_clean":"2024-04-23T21:22:23.622313+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:23.622313+0000","last_undegraded":"2024-04-23T21:22:23.622313+0000","last_fullsized":"2024-04-23T21:22:23.622313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:37:01.200688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043871600000000002,"stat_sum":{"num_bytes":87760278,"num_objects":78,"num_object_clones":21,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":32,"num_read":9736,"num_read_kb":56381,"num_write":868,"num_write_kb":32782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.324875+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:23.324875+0000","last_peered":"2024-04-23T21:22:23.324875+0000","last_clean":"2024-04-23T21:22:23.324875+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:23.324875+0000","last_undegraded":"2024-04-23T21:22:23.324875+0000","last_fullsized":"2024-04-23T21:22:23.324875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"79'472","reported_seq":4378,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.551391+0000","last_change":"2024-04-23T21:21:19.891842+0000","last_active":"2024-04-23T21:22:23.551391+0000","last_peered":"2024-04-23T21:22:23.551391+0000","last_clean":"2024-04-23T21:22:23.551391+0000","last_became_active":"2024-04-23T21:21:17.778907+0000","last_became_peered":"2024-04-23T21:21:17.778907+0000","last_unstale":"2024-04-23T21:22:23.551391+0000","last_undegraded":"2024-04-23T21:22:23.551391+0000","last_fullsized":"2024-04-23T21:22:23.551391+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:19.379323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074682100000000005,"stat_sum":{"num_bytes":55607103,"num_objects":47,"num_object_clones":8,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":22,"num_read":6797,"num_read_kb":14151,"num_write":623,"num_write_kb":7939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":138,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313087+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:23.313087+0000","last_peered":"2024-04-23T21:22:23.313087+0000","last_clean":"2024-04-23T21:22:23.313087+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:23.313087+0000","last_undegraded":"2024-04-23T21:22:23.313087+0000","last_fullsized":"2024-04-23T21:22:23.313087+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"77'203","reported_seq":2016,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379900+0000","last_change":"2024-04-23T21:21:49.999250+0000","last_active":"2024-04-23T21:22:23.379900+0000","last_peered":"2024-04-23T21:22:23.379900+0000","last_clean":"2024-04-23T21:22:23.379900+0000","last_became_active":"2024-04-23T21:21:49.944953+0000","last_became_peered":"2024-04-23T21:21:49.944953+0000","last_unstale":"2024-04-23T21:22:23.379900+0000","last_undegraded":"2024-04-23T21:22:23.379900+0000","last_fullsized":"2024-04-23T21:22:23.379900+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:50:55.721596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":30618672,"num_objects":64,"num_object_clones":18,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":33,"num_read":1933,"num_read_kb":14039,"num_write":219,"num_write_kb":10969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":3679788,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"78'260","reported_seq":6916,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.621270+0000","last_change":"2024-04-23T21:21:19.788510+0000","last_active":"2024-04-23T21:22:23.621270+0000","last_peered":"2024-04-23T21:22:23.621270+0000","last_clean":"2024-04-23T21:22:23.621270+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:23.621270+0000","last_undegraded":"2024-04-23T21:22:23.621270+0000","last_fullsized":"2024-04-23T21:22:23.621270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:11:20.878508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049684000000000002,"stat_sum":{"num_bytes":103445929,"num_objects":72,"num_object_clones":20,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":32,"num_read":6698,"num_read_kb":45882,"num_write":286,"num_write_kb":15594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"79'421","reported_seq":3283,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.411935+0000","last_change":"2024-04-23T21:21:19.892162+0000","last_active":"2024-04-23T21:22:23.411935+0000","last_peered":"2024-04-23T21:22:23.411935+0000","last_clean":"2024-04-23T21:22:23.411935+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:23.411935+0000","last_undegraded":"2024-04-23T21:22:23.411935+0000","last_fullsized":"2024-04-23T21:22:23.411935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:00:21.509463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077795399999999997,"stat_sum":{"num_bytes":40065512,"num_objects":58,"num_object_clones":10,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":28,"num_read":5262,"num_read_kb":12426,"num_write":422,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'365","reported_seq":2185,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.379776+0000","last_change":"2024-04-23T21:21:19.891570+0000","last_active":"2024-04-23T21:22:23.379776+0000","last_peered":"2024-04-23T21:22:23.379776+0000","last_clean":"2024-04-23T21:22:23.379776+0000","last_became_active":"2024-04-23T21:21:17.776323+0000","last_became_peered":"2024-04-23T21:21:17.776323+0000","last_unstale":"2024-04-23T21:22:23.379776+0000","last_undegraded":"2024-04-23T21:22:23.379776+0000","last_fullsized":"2024-04-23T21:22:23.379776+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:31:51.717415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057214699999999996,"stat_sum":{"num_bytes":61296972,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":26,"num_read":2554,"num_read_kb":25780,"num_write":404,"num_write_kb":12062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"79'280","reported_seq":1009,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.434041+0000","last_change":"2024-04-23T21:21:19.797766+0000","last_active":"2024-04-23T21:22:23.434041+0000","last_peered":"2024-04-23T21:22:23.434041+0000","last_clean":"2024-04-23T21:22:23.434041+0000","last_became_active":"2024-04-23T21:21:17.782105+0000","last_became_peered":"2024-04-23T21:21:17.782105+0000","last_unstale":"2024-04-23T21:22:23.434041+0000","last_undegraded":"2024-04-23T21:22:23.434041+0000","last_fullsized":"2024-04-23T21:22:23.434041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:59.575169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010469489999999999,"stat_sum":{"num_bytes":80032566,"num_objects":60,"num_object_clones":16,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":31,"num_read":1114,"num_read_kb":14927,"num_write":259,"num_write_kb":15541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"78'223","reported_seq":2447,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313316+0000","last_change":"2024-04-23T21:21:19.892379+0000","last_active":"2024-04-23T21:22:23.313316+0000","last_peered":"2024-04-23T21:22:23.313316+0000","last_clean":"2024-04-23T21:22:23.313316+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:23.313316+0000","last_undegraded":"2024-04-23T21:22:23.313316+0000","last_fullsized":"2024-04-23T21:22:23.313316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:21:06.090657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058441000000000005,"stat_sum":{"num_bytes":42361659,"num_objects":51,"num_object_clones":9,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":28,"num_read":2453,"num_read_kb":11389,"num_write":201,"num_write_kb":2716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"78'306","reported_seq":1208,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.313479+0000","last_change":"2024-04-23T21:21:19.892724+0000","last_active":"2024-04-23T21:22:23.313479+0000","last_peered":"2024-04-23T21:22:23.313479+0000","last_clean":"2024-04-23T21:22:23.313479+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:23.313479+0000","last_undegraded":"2024-04-23T21:22:23.313479+0000","last_fullsized":"2024-04-23T21:22:23.313479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:26:49.126009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066005100000000004,"stat_sum":{"num_bytes":36674923,"num_objects":44,"num_object_clones":7,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":23,"num_read":1797,"num_read_kb":8411,"num_write":357,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"79'854","reported_seq":9620,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.620002+0000","last_change":"2024-04-23T21:21:19.892740+0000","last_active":"2024-04-23T21:22:23.620002+0000","last_peered":"2024-04-23T21:22:23.620002+0000","last_clean":"2024-04-23T21:22:23.620002+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:23.620002+0000","last_undegraded":"2024-04-23T21:22:23.620002+0000","last_fullsized":"2024-04-23T21:22:23.620002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:26:22.317940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053108400000000003,"stat_sum":{"num_bytes":118035912,"num_objects":75,"num_object_clones":16,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":30,"num_read":12882,"num_read_kb":57742,"num_write":1235,"num_write_kb":15540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"79'359","reported_seq":2571,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.300337+0000","last_change":"2024-04-23T21:21:19.891893+0000","last_active":"2024-04-23T21:22:23.300337+0000","last_peered":"2024-04-23T21:22:23.300337+0000","last_clean":"2024-04-23T21:22:23.300337+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:23.300337+0000","last_undegraded":"2024-04-23T21:22:23.300337+0000","last_fullsized":"2024-04-23T21:22:23.300337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:19:15.735525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059656900000000003,"stat_sum":{"num_bytes":97089947,"num_objects":68,"num_object_clones":16,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":30,"num_read":3001,"num_read_kb":23044,"num_write":430,"num_write_kb":16380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"79'233","reported_seq":2069,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.437389+0000","last_change":"2024-04-23T21:21:19.892383+0000","last_active":"2024-04-23T21:22:23.437389+0000","last_peered":"2024-04-23T21:22:23.437389+0000","last_clean":"2024-04-23T21:22:23.437389+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:23.437389+0000","last_undegraded":"2024-04-23T21:22:23.437389+0000","last_fullsized":"2024-04-23T21:22:23.437389+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:14:25.000275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067082800000000005,"stat_sum":{"num_bytes":87772664,"num_objects":60,"num_object_clones":15,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":25,"num_read":1790,"num_read_kb":20042,"num_write":268,"num_write_kb":14124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"79'473","reported_seq":2343,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.415895+0000","last_change":"2024-04-23T21:21:32.840236+0000","last_active":"2024-04-23T21:22:23.415895+0000","last_peered":"2024-04-23T21:22:23.415895+0000","last_clean":"2024-04-23T21:22:23.415895+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:23.415895+0000","last_undegraded":"2024-04-23T21:22:23.415895+0000","last_fullsized":"2024-04-23T21:22:23.415895+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:59:39.713723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048511600000000001,"stat_sum":{"num_bytes":72590514,"num_objects":66,"num_object_clones":10,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":26,"num_read":3829,"num_read_kb":36394,"num_write":610,"num_write_kb":19593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"79'797","reported_seq":2637,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-04-23T21:22:23.552700+0000","last_change":"2024-04-23T21:21:19.891960+0000","last_active":"2024-04-23T21:22:23.552700+0000","last_peered":"2024-04-23T21:22:23.552700+0000","last_clean":"2024-04-23T21:22:23.552700+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:23.552700+0000","last_undegraded":"2024-04-23T21:22:23.552700+0000","last_fullsized":"2024-04-23T21:22:23.552700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:47:19.290204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000437358,"stat_sum":{"num_bytes":83105146,"num_objects":70,"num_object_clones":14,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":5122,"num_read_kb":38642,"num_write":1229,"num_write_kb":23791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2224873740,"num_objects":2023,"num_object_clones":455,"num_object_copies":4046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":954,"num_read":158054,"num_read_kb":900099,"num_write":15438,"num_write_kb":537839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":7491189,"num_keys_recovered":59,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":84684800,"data_stored":843464538,"data_compressed":50229476,"data_compressed_allocated":56201216,"data_compressed_original":818171312,"omap_allocated":0,"internal_metadata":0},"log_size":12663,"ondisk_log_size":12663,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58284,"kb_used_data":13220,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94313556,"statfs":{"total":96636764160,"available":96577081344,"internally_reserved":0,"allocated":13537280,"data_stored":109891485,"data_compressed":7373241,"data_compressed_allocated":8323072,"data_compressed_original":105761647,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35736,"kb_used_data":4552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336104,"statfs":{"total":96636764160,"available":96600170496,"internally_reserved":0,"allocated":4661248,"data_stored":39866008,"data_compressed":2093626,"data_compressed_allocated":2482176,"data_compressed_original":38250387,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48660,"kb_used_data":9152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94323180,"statfs":{"total":96636764160,"available":96586936320,"internally_reserved":0,"allocated":9371648,"data_stored":82139560,"data_compressed":4627598,"data_compressed_allocated":5382144,"data_compressed_original":79023297,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70912,"kb_used_data":21688,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94300928,"statfs":{"total":96636764160,"available":96564150272,"internally_reserved":0,"allocated":22208512,"data_stored":233826812,"data_compressed":13755609,"data_compressed_allocated":14831616,"data_compressed_original":227589506,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54612,"kb_used_data":10952,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94317228,"statfs":{"total":96636764160,"available":96580841472,"internally_reserved":0,"allocated":11214848,"data_stored":88298549,"data_compressed":4967631,"data_compressed_allocated":5799936,"data_compressed_original":83783582,"omap_allocated":1593,"internal_metadata":44693959},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36044,"kb_used_data":4864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335796,"statfs":{"total":96636764160,"available":96599855104,"internally_reserved":0,"allocated":4980736,"data_stored":36036521,"data_compressed":2125865,"data_compressed_allocated":2441216,"data_compressed_original":34212278,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528328,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40852,"kb_used_data":8332,"kb_used_omap":6,"kb_used_meta":32505,"kb_avail":94330988,"statfs":{"total":96636764160,"available":96594931712,"internally_reserved":0,"allocated":8531968,"data_stored":87139627,"data_compressed":5294755,"data_compressed_allocated":5435392,"data_compressed_original":84836026,"omap_allocated":6748,"internal_metadata":33285540},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44132,"kb_used_data":17116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327708,"statfs":{"total":96636764160,"available":96591572992,"internally_reserved":0,"allocated":17526784,"data_stored":170867382,"data_compressed":10067507,"data_compressed_allocated":11669504,"data_compressed_original":166026987,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16613376,"data_stored":170539900,"data_compressed":10076774,"data_compressed_allocated":11669504,"data_compressed_original":166125291,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7561216,"data_stored":85500553,"data_compressed":5196367,"data_compressed_allocated":5316608,"data_compressed_original":83566266,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":35604935,"data_compressed":2124504,"data_compressed_allocated":2437120,"data_compressed_original":34183606,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10301440,"data_stored":88010323,"data_compressed":4984973,"data_compressed_allocated":5799936,"data_compressed_original":83921168,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":21295104,"data_stored":233499325,"data_compressed":13764886,"data_compressed_allocated":14831616,"data_compressed_original":227687810,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8454144,"data_stored":81877572,"data_compressed":4643053,"data_compressed_allocated":5382144,"data_compressed_original":79187137,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3743744,"data_stored":39440180,"data_compressed":2093626,"data_compressed_allocated":2482176,"data_compressed_original":38250387,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12632064,"data_stored":108991750,"data_compressed":7345293,"data_compressed_allocated":8282112,"data_compressed_original":105249647,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:28.174 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-23T21:22:28.174 INFO:teuthology.orchestra.run.smithi111.stdout:5295 writesame 0x76b3e42 thru 0x76c3707 data_size 0x7(0xf8c6 bytes) 2024-04-23T21:22:28.174 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:28.174 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-23T21:22:28.175 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-04-23T21:22:28.179 INFO:teuthology.orchestra.run.smithi111.stdout:5298 read 0x4d48e1 thru 0x4d590a (0x102a bytes) 2024-04-23T21:22:28.180 INFO:teuthology.orchestra.run.smithi111.stdout:5300 read 0x6ba74e0 thru 0x6baaa11 (0x3532 bytes) 2024-04-23T21:22:28.180 INFO:teuthology.orchestra.run.smithi111.stdout:5301 write 0x7c3174d thru 0x7c3da25 (0xc2d9 bytes) 2024-04-23T21:22:28.181 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:28.181 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:28.183 INFO:teuthology.orchestra.run.smithi111.stdout:5302 writesame 0x1947a05 thru 0x194b5b4 data_size 0x8(0x3bb0 bytes) 2024-04-23T21:22:28.187 INFO:teuthology.orchestra.run.smithi111.stdout:5303 write 0x3270729 thru 0x327e12c (0xda04 bytes) 2024-04-23T21:22:28.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:28.191 INFO:teuthology.orchestra.run.smithi111.stdout:5305 read 0x30d6968 thru 0x30e258f (0xbc28 bytes) 2024-04-23T21:22:28.192 INFO:teuthology.orchestra.run.smithi111.stdout:5306 read 0x54a6c27 thru 0x54ac1b2 (0x558c bytes) 2024-04-23T21:22:28.192 INFO:teuthology.orchestra.run.smithi111.stdout:5307 write 0x865844 thru 0x8695ae (0x3d6b bytes) 2024-04-23T21:22:28.192 INFO:teuthology.orchestra.run.smithi111.stdout:5308 writesame 0x97d84e thru 0x989bcc data_size 0x1(0xc37f bytes) 2024-04-23T21:22:28.216 INFO:teuthology.orchestra.run.smithi111.stdout:5309 write 0x96752fc thru 0x9681d73 (0xca78 bytes) 2024-04-23T21:22:28.222 INFO:teuthology.orchestra.run.smithi111.stdout:5310 write 0x754d03a thru 0x7551fdd (0x4fa4 bytes) 2024-04-23T21:22:28.226 INFO:teuthology.orchestra.run.smithi111.stdout:5311 write 0x4267041 thru 0x4272dd5 (0xbd95 bytes) 2024-04-23T21:22:28.230 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.230 INFO:teuthology.orchestra.run.smithi111.stdout:5313 read 0xb9f034 thru 0xba3ce3 (0x4cb0 bytes) 2024-04-23T21:22:28.230 INFO:teuthology.orchestra.run.smithi111.stdout:5314 writesame 0xe72f720 thru 0xe73062a data_size 0x1(0xf0b bytes) 2024-04-23T21:22:28.247 INFO:teuthology.orchestra.run.smithi111.stdout:5315 trunc from 0xeade3b0 to 0x2337bf8 2024-04-23T21:22:28.265 INFO:teuthology.orchestra.run.smithi111.stdout:5316 read 0x1d42206 thru 0x1d46eda (0x4cd5 bytes) 2024-04-23T21:22:28.265 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.275 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:28.276+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:28.279 INFO:teuthology.orchestra.run.smithi111.stdout:5319 write 0x9141bae thru 0x914425a (0x26ad bytes) 2024-04-23T21:22:28.282 INFO:teuthology.orchestra.run.smithi111.stdout:5321 read 0xe0d85c thru 0xe0dd16 (0x4bb bytes) 2024-04-23T21:22:28.282 INFO:teuthology.orchestra.run.smithi111.stdout:5322 writesame 0x5f57732 thru 0x5f66033 data_size 0x19(0xe902 bytes) 2024-04-23T21:22:28.286 INFO:teuthology.orchestra.run.smithi111.stdout:5324 write 0x54cbc84 thru 0x54d7ce2 (0xc05f bytes) 2024-04-23T21:22:28.290 INFO:teuthology.orchestra.run.smithi111.stdout:5325 read 0x1f9f7b9 thru 0x1fad836 (0xe07e bytes) 2024-04-23T21:22:28.290 INFO:teuthology.orchestra.run.smithi111.stdout:5326 punch from 0x3873ec5 to 0x387f306, (0xb441 bytes) 2024-04-23T21:22:28.300 INFO:teuthology.orchestra.run.smithi111.stdout:5327 write 0xd210210 thru 0xd21ccda (0xcacb bytes) 2024-04-23T21:22:28.304 INFO:teuthology.orchestra.run.smithi111.stdout:5331 write 0x6377ca1 thru 0x6385125 (0xd485 bytes) 2024-04-23T21:22:28.309 INFO:teuthology.orchestra.run.smithi111.stdout:5332 write 0x6b090d2 thru 0x6b16714 (0xd643 bytes) 2024-04-23T21:22:28.312 INFO:teuthology.orchestra.run.smithi111.stdout:5334 writesame 0xa68aa14 thru 0xa693f51 data_size 0x7(0x953e bytes) 2024-04-23T21:22:28.316 INFO:teuthology.orchestra.run.smithi111.stdout:5335 clone 46 order 24 su 1048576 sc 6 2024-04-23T21:22:28.345 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:28.345 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:28.345 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:28.346 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:28.346 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:28.346 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:22:28.355 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:28.390 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:28.390+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:28.463 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:28.464+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:28.498 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:22:28.498+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e84 definitely_dead 0 2024-04-23T21:22:28.500 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:28.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544238131, 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:28.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544200011000001, 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.505 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:28.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544177611999999, 2024-04-23T21:22:28.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:28.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:28.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:28.514 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.515 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544158216, 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:28.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:28.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544138617999998, 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.523 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:28.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:28.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544118404999999, 2024-04-23T21:22:28.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:28.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.531 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544086239000002, 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:28.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544061626000001, 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544040627999998, 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.541 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:28.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:28.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.544020328999999, 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:28.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:28.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.543942540000003, 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.550 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:28.554 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.542447166000002, 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:28.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:28.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:28.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.542397276000003, 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.559 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:28.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.542373507000001, 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.564 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:28.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:28.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.568 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.542352501000003, 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:28.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.542332895999998, 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.526330747000003, 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.526312236000003, 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.526291790999998, 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 36.547214578999998, 2024-04-23T21:22:28.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:28.616 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:28.619 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #44, image image_client.0-clone44 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent45 2024-04-23T21:22:28.619 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.620 INFO:teuthology.orchestra.run.smithi111.stdout:5338 write 0x355d6f thru 0x35a2a1 (0x4533 bytes) 2024-04-23T21:22:28.620 INFO:teuthology.orchestra.run.smithi111.stdout:5340 read 0x387ef48 thru 0x38872e1 (0x839a bytes) 2024-04-23T21:22:28.620 INFO:teuthology.orchestra.run.smithi111.stdout:5342 writesame 0x1340748 thru 0x1346202 data_size 0x1(0x5abb bytes) 2024-04-23T21:22:28.620 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:28.618+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:28.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:28.632 INFO:teuthology.orchestra.run.smithi111.stdout:5343 write 0x513a4f6 thru 0x5145581 (0xb08c bytes) 2024-04-23T21:22:28.633 INFO:teuthology.orchestra.run.smithi111.stdout:5345 read 0xd02674c thru 0xd031f2c (0xb7e1 bytes) 2024-04-23T21:22:28.633 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.634 INFO:teuthology.orchestra.run.smithi111.stdout:5347 write 0x9a15c14 thru 0x9a21341 (0xb72e bytes) 2024-04-23T21:22:28.638 INFO:teuthology.orchestra.run.smithi111.stdout:5348 write 0x86a35c6 thru 0x86aeac1 (0xb4fc bytes) 2024-04-23T21:22:28.641 INFO:teuthology.orchestra.run.smithi111.stdout:5349 punch from 0xb7bf8b4 to 0xb7cbd48, (0xc494 bytes) 2024-04-23T21:22:28.651 INFO:teuthology.orchestra.run.smithi111.stdout:5350 trunc from 0xd21ccdb to 0x566bbee 2024-04-23T21:22:28.691 INFO:teuthology.orchestra.run.smithi111.stdout:5351 write 0xe54ae2a thru 0xe54de5c (0x3033 bytes) 2024-04-23T21:22:28.693 INFO:teuthology.orchestra.run.smithi111.stdout:5353 write 0x995a164 thru 0x9966b62 (0xc9ff bytes) 2024-04-23T21:22:28.697 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.697 INFO:teuthology.orchestra.run.smithi111.stdout:5355 write 0x5c3ec71 thru 0x5c43e5d (0x51ed bytes) 2024-04-23T21:22:28.698 INFO:teuthology.orchestra.run.smithi111.stdout:5356 write 0x3f84c14 thru 0x3f89b3a (0x4f27 bytes) 2024-04-23T21:22:28.700 INFO:teuthology.orchestra.run.smithi111.stdout:5357 punch from 0x2d87fef to 0x2d913d0, (0x93e1 bytes) 2024-04-23T21:22:28.702 INFO:teuthology.orchestra.run.smithi111.stdout:5358 trunc from 0xe54de5d to 0xcaf9ace 2024-04-23T21:22:28.711 INFO:teuthology.orchestra.run.smithi111.stdout:5363 trunc from 0xcaf9ace to 0xe0f5b2c 2024-04-23T21:22:28.717 INFO:teuthology.orchestra.run.smithi111.stdout:5364 read 0x5827f57 thru 0x582e0ff (0x61a9 bytes) 2024-04-23T21:22:28.718 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.718 INFO:teuthology.orchestra.run.smithi111.stdout:5366 read 0xc01e1bd thru 0xc0271fb (0x903f bytes) 2024-04-23T21:22:28.718 INFO:teuthology.orchestra.run.smithi111.stdout:5367 write 0xa05d9e2 thru 0xa063e3f (0x645e bytes) 2024-04-23T21:22:28.718 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:28.719+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:28.721 INFO:teuthology.orchestra.run.smithi111.stdout:5369 read 0x61c2690 thru 0x61cccb2 (0xa623 bytes) 2024-04-23T21:22:28.721 INFO:teuthology.orchestra.run.smithi111.stdout:5371 read 0xaea2db1 thru 0xaea4e1d (0x206d bytes) 2024-04-23T21:22:28.721 INFO:teuthology.orchestra.run.smithi111.stdout:5372 write 0x91a797e thru 0x91add48 (0x63cb bytes) 2024-04-23T21:22:28.731 INFO:teuthology.orchestra.run.smithi111.stdout:5373 trunc from 0xe0f5b2c to 0x8d236b7 2024-04-23T21:22:28.742 INFO:teuthology.orchestra.run.smithi111.stdout:5374 writesame 0x89dc337 thru 0x89e38b8 data_size 0x1a(0x7582 bytes) 2024-04-23T21:22:28.745 INFO:teuthology.orchestra.run.smithi111.stdout:5376 punch from 0x4bd519a to 0x4bde712, (0x9578 bytes) 2024-04-23T21:22:28.745 INFO:teuthology.orchestra.run.smithi111.stdout:5377 writesame 0x1f116a3 thru 0x1f1e715 data_size 0x2b(0xd073 bytes) 2024-04-23T21:22:28.759 INFO:teuthology.orchestra.run.smithi111.stdout:5379 writesame 0xd333981 thru 0xd33e311 data_size 0x1(0xa991 bytes) 2024-04-23T21:22:28.759 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.760 INFO:teuthology.orchestra.run.smithi111.stdout:5381 trunc from 0xd33e312 to 0xd9fe7c3 2024-04-23T21:22:28.765 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.765 INFO:teuthology.orchestra.run.smithi111.stdout:5383 read 0x436c3ec thru 0x43749db (0x85f0 bytes) 2024-04-23T21:22:28.766 INFO:teuthology.orchestra.run.smithi111.stdout:5384 write 0x303df18 thru 0x3049eb8 (0xbfa1 bytes) 2024-04-23T21:22:28.767 INFO:teuthology.orchestra.run.smithi111.stdout:5385 write 0xd6bacfd thru 0xd6bf95c (0x4c60 bytes) 2024-04-23T21:22:28.770 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:28.770 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:28.770 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:28.770 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:28.770 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.771 INFO:teuthology.orchestra.run.smithi111.stdout:5390 punch from 0x1e6b9b7 to 0x1e6ee81, (0x34ca bytes) 2024-04-23T21:22:28.771 INFO:teuthology.orchestra.run.smithi111.stdout:5391 write 0x35434a7 thru 0x3544b4e (0x16a8 bytes) 2024-04-23T21:22:28.772 INFO:teuthology.orchestra.run.smithi111.stdout:5392 write 0x6849f0c thru 0x685451c (0xa611 bytes) 2024-04-23T21:22:28.775 INFO:teuthology.orchestra.run.smithi111.stdout:5394 read 0x285459 thru 0x287c4e (0x27f6 bytes) 2024-04-23T21:22:28.775 INFO:teuthology.orchestra.run.smithi111.stdout:5395 writesame 0x637aafa thru 0x6382dba data_size 0x11(0x82c1 bytes) 2024-04-23T21:22:28.778 INFO:teuthology.orchestra.run.smithi111.stdout:5397 read 0x8a4130e thru 0x8a48d72 (0x7a65 bytes) 2024-04-23T21:22:28.778 INFO:teuthology.orchestra.run.smithi111.stdout:5398 read 0xa9e45c8 thru 0xa9ebebc (0x78f5 bytes) 2024-04-23T21:22:28.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:28.793 INFO:teuthology.orchestra.run.smithi111.stdout:5401 trunc from 0xd9fe7c3 to 0xe2acb4 2024-04-23T21:22:28.808 INFO:teuthology.orchestra.run.smithi111.stdout:5403 write 0xc7cc34 thru 0xc84bc0 (0x7f8d bytes) 2024-04-23T21:22:28.815 INFO:teuthology.orchestra.run.smithi111.stdout:5406 trunc from 0xe2acb4 to 0x5b03c62 2024-04-23T21:22:28.819 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:28.819+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:28.821 INFO:teuthology.orchestra.run.smithi111.stdout:5408 read 0x287b8ed thru 0x287ed4e (0x3462 bytes) 2024-04-23T21:22:28.822 INFO:teuthology.orchestra.run.smithi111.stdout:5409 punch from 0x9c555a to 0x9d45bb, (0xf061 bytes) 2024-04-23T21:22:28.822 INFO:teuthology.orchestra.run.smithi111.stdout:5410 write 0x2ffa6b thru 0x304877 (0x4e0d bytes) 2024-04-23T21:22:28.823 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.823 INFO:teuthology.orchestra.run.smithi111.stdout:5412 write 0x33bb80d thru 0x33c52ae (0x9aa2 bytes) 2024-04-23T21:22:28.824 INFO:teuthology.orchestra.run.smithi111.stdout:5414 writesame 0x38166dd thru 0x381c68a data_size 0x25(0x5fae bytes) 2024-04-23T21:22:28.826 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.826 INFO:teuthology.orchestra.run.smithi111.stdout:5417 read 0x209f15 thru 0x217d39 (0xde25 bytes) 2024-04-23T21:22:28.827 INFO:teuthology.orchestra.run.smithi111.stdout:5418 punch from 0x113fa66 to 0x11444e2, (0x4a7c bytes) 2024-04-23T21:22:28.837 INFO:teuthology.orchestra.run.smithi111.stdout:5419 write 0x9f160bb thru 0x9f18a23 (0x2969 bytes) 2024-04-23T21:22:28.847 INFO:teuthology.orchestra.run.smithi111.stdout:5420 trunc from 0x9f18a24 to 0x49f6fb7 2024-04-23T21:22:28.868 INFO:teuthology.orchestra.run.smithi111.stdout:5421 write 0x75e12bf thru 0x75e6fb8 (0x5cfa bytes) 2024-04-23T21:22:28.884 INFO:teuthology.orchestra.run.smithi111.stdout:5422 write 0xd2cb422 thru 0xd2ce107 (0x2ce6 bytes) 2024-04-23T21:22:28.887 INFO:teuthology.orchestra.run.smithi111.stdout:5423 write 0x7dea5a7 thru 0x7dee268 (0x3cc2 bytes) 2024-04-23T21:22:28.890 INFO:teuthology.orchestra.run.smithi111.stdout:5424 punch from 0x39edac8 to 0x39f3a13, (0x5f4b bytes) 2024-04-23T21:22:28.890 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.890 INFO:teuthology.orchestra.run.smithi111.stdout:5428 write 0x3a5a861 thru 0x3a625c8 (0x7d68 bytes) 2024-04-23T21:22:28.895 INFO:teuthology.orchestra.run.smithi111.stdout:5429 punch from 0x3eee184 to 0x3efdef4, (0xfd70 bytes) 2024-04-23T21:22:28.895 INFO:teuthology.orchestra.run.smithi111.stdout:5431 write 0xb5757be thru 0xb57764c (0x1e8f bytes) 2024-04-23T21:22:28.896 INFO:teuthology.orchestra.run.smithi111.stdout:5432 write 0x2249fc thru 0x231452 (0xca57 bytes) 2024-04-23T21:22:28.896 INFO:teuthology.orchestra.run.smithi111.stdout:5434 write 0x6a36cca thru 0x6a455fa (0xe931 bytes) 2024-04-23T21:22:28.899 INFO:teuthology.orchestra.run.smithi111.stdout:5435 write 0xb3246c thru 0xb33142 (0xcd7 bytes) 2024-04-23T21:22:28.900 INFO:teuthology.orchestra.run.smithi111.stdout:5436 read 0xa7ebd07 thru 0xa7f73d2 (0xb6cc bytes) 2024-04-23T21:22:28.901 INFO:teuthology.orchestra.run.smithi111.stdout:5437 read 0xa9b139b thru 0xa9b3e48 (0x2aae bytes) 2024-04-23T21:22:28.901 INFO:teuthology.orchestra.run.smithi111.stdout:5438 write 0x384b023 thru 0x3858f4d (0xdf2b bytes) 2024-04-23T21:22:28.903 INFO:teuthology.orchestra.run.smithi111.stdout:5439 punch from 0x7a8ee36 to 0x7a8ef9b, (0x165 bytes) 2024-04-23T21:22:28.904 INFO:teuthology.orchestra.run.smithi111.stdout:5440 trunc from 0xd2ce108 to 0xde26119 2024-04-23T21:22:28.909 INFO:teuthology.orchestra.run.smithi111.stdout:5441 read 0x565a6be thru 0x565f8ca (0x520d bytes) 2024-04-23T21:22:28.910 INFO:teuthology.orchestra.run.smithi111.stdout:5443 write 0x56a4d64 thru 0x56a5602 (0x89f bytes) 2024-04-23T21:22:28.911 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xedec0d8 2024-04-23T21:22:28.912 INFO:teuthology.orchestra.run.smithi111.stdout:5444 trunc from 0xde26119 to 0xedec0d8 2024-04-23T21:22:28.918 INFO:teuthology.orchestra.run.smithi111.stdout:5446 writesame 0x75d4450 thru 0x75ddb8f data_size 0x79(0x9740 bytes) 2024-04-23T21:22:28.919 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:28.920+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:28.920 INFO:teuthology.orchestra.run.smithi111.stdout:5447 read 0x1d56598 thru 0x1d5a7f7 (0x4260 bytes) 2024-04-23T21:22:28.920 INFO:teuthology.orchestra.run.smithi111.stdout:5448 write 0xdf1a722 thru 0xdf29d51 (0xf630 bytes) 2024-04-23T21:22:28.931 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:28.931 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:28.931 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:28.932 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:28.932 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:28.935 INFO:teuthology.orchestra.run.smithi111.stdout:5451 trunc from 0xedec0d8 to 0x5f062f7 2024-04-23T21:22:28.941 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:28.949 INFO:teuthology.orchestra.run.smithi111.stdout:5452 read 0x40f1a38 thru 0x4100ad4 (0xf09d bytes) 2024-04-23T21:22:28.949 INFO:teuthology.orchestra.run.smithi111.stdout:5453 write 0x9bb099 thru 0x9bf3a5 (0x430d bytes) 2024-04-23T21:22:28.951 INFO:teuthology.orchestra.run.smithi111.stdout:5454 write 0x1505839 thru 0x1508196 (0x295e bytes) 2024-04-23T21:22:28.952 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.952 INFO:teuthology.orchestra.run.smithi111.stdout:5457 punch from 0x32dadca to 0x32dd430, (0x2666 bytes) 2024-04-23T21:22:28.953 INFO:teuthology.orchestra.run.smithi111.stdout:5458 read 0x2f43e76 thru 0x2f4b565 (0x76f0 bytes) 2024-04-23T21:22:28.953 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.953 INFO:teuthology.orchestra.run.smithi111.stdout:5461 read 0x357f744 thru 0x358beeb (0xc7a8 bytes) 2024-04-23T21:22:28.965 INFO:teuthology.orchestra.run.smithi111.stdout:5462 write 0xb07d822 thru 0xb08d794 (0xff73 bytes) 2024-04-23T21:22:28.977 INFO:teuthology.orchestra.run.smithi111.stdout:5463 write 0xd5af402 thru 0xd5b43d9 (0x4fd8 bytes) 2024-04-23T21:22:28.980 INFO:teuthology.orchestra.run.smithi111.stdout:5464 punch from 0x37b5c59 to 0x37c19c5, (0xbd6c bytes) 2024-04-23T21:22:28.980 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.980 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:28.980 INFO:teuthology.orchestra.run.smithi111.stdout:5469 punch from 0x263c148 to 0x264afdf, (0xee97 bytes) 2024-04-23T21:22:28.986 INFO:teuthology.orchestra.run.smithi111.stdout:5470 writesame 0xe0cf6b2 thru 0xe0da928 data_size 0x61(0xb277 bytes) 2024-04-23T21:22:28.993 INFO:teuthology.orchestra.run.smithi111.stdout:5472 trunc from 0xe0da929 to 0x7efee01 2024-04-23T21:22:29.007 INFO:teuthology.orchestra.run.smithi111.stdout:5474 read 0x1fba71c thru 0x1fc19b7 (0x729c bytes) 2024-04-23T21:22:29.008 INFO:teuthology.orchestra.run.smithi111.stdout:5475 writesame 0x30d8d7f thru 0x30e6751 data_size 0x1(0xd9d3 bytes) 2024-04-23T21:22:29.020 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:29.020+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:29.031 INFO:teuthology.orchestra.run.smithi111.stdout:5476 punch from 0xc916b2 to 0xc921ab, (0xaf9 bytes) 2024-04-23T21:22:29.031 INFO:teuthology.orchestra.run.smithi111.stdout:5477 write 0x1a27e05 thru 0x1a37b88 (0xfd84 bytes) 2024-04-23T21:22:29.033 INFO:teuthology.orchestra.run.smithi111.stdout:5479 punch from 0x2f53cd4 to 0x2f5a67c, (0x69a8 bytes) 2024-04-23T21:22:29.033 INFO:teuthology.orchestra.run.smithi111.stdout:5480 punch from 0x1697563 to 0x16a69e5, (0xf482 bytes) 2024-04-23T21:22:29.033 INFO:teuthology.orchestra.run.smithi111.stdout:5481 write 0x52f95ce thru 0x52ff669 (0x609c bytes) 2024-04-23T21:22:29.036 INFO:teuthology.orchestra.run.smithi111.stdout:5483 read 0x6b9e235 thru 0x6ba4434 (0x6200 bytes) 2024-04-23T21:22:29.036 INFO:teuthology.orchestra.run.smithi111.stdout:5486 writesame 0xaeeffa thru 0xafddb9 data_size 0x60(0xedc0 bytes) 2024-04-23T21:22:29.037 INFO:teuthology.orchestra.run.smithi111.stdout:5488 writesame 0x775ff86 thru 0x7762b7a data_size 0x21(0x2bf5 bytes) 2024-04-23T21:22:29.041 INFO:teuthology.orchestra.run.smithi111.stdout:5489 write 0x50ebc7c thru 0x50fa350 (0xe6d5 bytes) 2024-04-23T21:22:29.043 INFO:teuthology.orchestra.run.smithi111.stdout:5490 write 0x5584060 thru 0x559364b (0xf5ec bytes) 2024-04-23T21:22:29.047 INFO:teuthology.orchestra.run.smithi111.stdout:5492 trunc from 0x7efee01 to 0xa2e44e3 2024-04-23T21:22:29.054 INFO:teuthology.orchestra.run.smithi111.stdout:5493 trunc from 0xa2e44e3 to 0xb347e12 2024-04-23T21:22:29.060 INFO:teuthology.orchestra.run.smithi111.stdout:5494 punch from 0x1afd21a to 0x1afe8b3, (0x1699 bytes) 2024-04-23T21:22:29.068 INFO:teuthology.orchestra.run.smithi111.stdout:5495 write 0xcdaa19e thru 0xcdb9737 (0xf59a bytes) 2024-04-23T21:22:29.073 INFO:teuthology.orchestra.run.smithi111.stdout:5496 read 0x3ad3605 thru 0x3ad6d21 (0x371d bytes) 2024-04-23T21:22:29.073 INFO:teuthology.orchestra.run.smithi111.stdout:5497 punch from 0x88c2110 to 0x88c8a01, (0x68f1 bytes) 2024-04-23T21:22:29.073 INFO:teuthology.orchestra.run.smithi111.stdout:5498 read 0x32c8483 thru 0x32d1b27 (0x96a5 bytes) 2024-04-23T21:22:29.074 INFO:teuthology.orchestra.run.smithi111.stdout:5499 writesame 0x6842e89 thru 0x684f8bb data_size 0x1(0xca33 bytes) 2024-04-23T21:22:29.090 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:29.090 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:29.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:29.090 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.2141287490000003, 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.091 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:29.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1477102400000003, 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.097 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:29.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:29.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:29.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1476928589999993, 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.105 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:29.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1476753019999997, 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:29.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.114 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1460151070000002, 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:29.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:29.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1459932110000004, 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.123 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1459658610000005, 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:29.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.128 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1459417730000006, 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.132 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1459107139999993, 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.133 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:29.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:29.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:29.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1447106460000001, 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1446882130000002, 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.142 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:29.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1446646250000008, 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.147 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:29.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:29.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1446446859999995, 2024-04-23T21:22:29.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:29.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1446162110000007, 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.156 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:29.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:29.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1432334669999999, 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:29.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:29.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1381368429999998, 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.170 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.171 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:29.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1377902169999992, 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:29.178 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.182 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1262753130000007, 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:29.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1262553739999994, 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:29.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:29.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 9.1249551029999996, 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.191 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:29.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.196 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:29.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:29.198 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:29.197+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:29.199 INFO:teuthology.orchestra.run.smithi111.stdout:5500 trunc from 0xcdb9738 to 0x41b6b8a 2024-04-23T21:22:29.199 INFO:teuthology.orchestra.run.smithi111.stdout:5501 write 0x18634d9 thru 0x186f0b5 (0xbbdd bytes) 2024-04-23T21:22:29.199 INFO:teuthology.orchestra.run.smithi111.stdout:5502 writesame 0x81d9a19 thru 0x81e7e23 data_size 0x1(0xe40b bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5503 read 0x2f1e873 thru 0x2f255a0 (0x6d2e bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5505 read 0x6033df3 thru 0x60411b4 (0xd3c2 bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5509 write 0xb8301f9 thru 0xb833859 (0x3661 bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5510 trunc from 0xb83385a to 0x4b16e29 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5512 write 0xdfa5c8d thru 0xdfb0897 (0xac0b bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5514 read 0x56aafc thru 0x579bb7 (0xf0bc bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5516 punch from 0x16e8bad to 0x16ed3bf, (0x4812 bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5517 punch from 0xbdbd774 to 0xbdc9b86, (0xc412 bytes) 2024-04-23T21:22:29.200 INFO:teuthology.orchestra.run.smithi111.stdout:5518 write 0xe10eafe thru 0xe11ad60 (0xc263 bytes) 2024-04-23T21:22:29.203 INFO:teuthology.orchestra.run.smithi111.stdout:5519 trunc from 0xe11ad61 to 0xcfd0163 2024-04-23T21:22:29.213 INFO:teuthology.orchestra.run.smithi111.stdout:5520 write 0x75dd66a thru 0x75ec40f (0xeda6 bytes) 2024-04-23T21:22:29.216 INFO:teuthology.orchestra.run.smithi111.stdout:5522 read 0x3a23c63 thru 0x3a312ea (0xd688 bytes) 2024-04-23T21:22:29.216 INFO:teuthology.orchestra.run.smithi111.stdout:5524 read 0xcda5633 thru 0xcdaa314 (0x4ce2 bytes) 2024-04-23T21:22:29.216 INFO:teuthology.orchestra.run.smithi111.stdout:5526 write 0x1e1ae9f thru 0x1e1c61d (0x177f bytes) 2024-04-23T21:22:29.224 INFO:teuthology.orchestra.run.smithi111.stdout:5528 trunc from 0xcfd0163 to 0x7d4e466 2024-04-23T21:22:29.240 INFO:teuthology.orchestra.run.smithi111.stdout:5529 trunc from 0x7d4e466 to 0xb7a6a1b 2024-04-23T21:22:29.245 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:29.249 INFO:teuthology.orchestra.run.smithi111.stdout:5532 trunc from 0xb7a6a1b to 0xe618f6a 2024-04-23T21:22:29.255 INFO:teuthology.orchestra.run.smithi111.stdout:5533 read 0xba7b3c1 thru 0xba8507a (0x9cba bytes) 2024-04-23T21:22:29.255 INFO:teuthology.orchestra.run.smithi111.stdout:5539 read 0x101eb4a thru 0x1024db0 (0x6267 bytes) 2024-04-23T21:22:29.269 INFO:teuthology.orchestra.run.smithi111.stdout:5540 trunc from 0xe618f6a to 0x373cf3f 2024-04-23T21:22:29.296 INFO:teuthology.orchestra.run.smithi111.stdout:5541 writesame 0xa887ef6 thru 0xa889822 data_size 0x1(0x192d bytes) 2024-04-23T21:22:29.298 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:29.299+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:29.302 INFO:teuthology.orchestra.run.smithi111.stdout:5542 writesame 0xababf60 thru 0xabb3500 data_size 0x1(0x75a1 bytes) 2024-04-23T21:22:29.302 INFO:teuthology.orchestra.run.smithi111.stdout:5543 write 0x33427aa thru 0x3346b29 (0x4380 bytes) 2024-04-23T21:22:29.306 INFO:teuthology.orchestra.run.smithi111.stdout:5544 trunc from 0xabb3501 to 0xcd067ce 2024-04-23T21:22:29.312 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:29.323 INFO:teuthology.orchestra.run.smithi111.stdout:5546 trunc from 0xcd067ce to 0x3bc3020 2024-04-23T21:22:29.330 INFO:teuthology.orchestra.run.smithi111.stdout:5547 read 0x7c9822 thru 0x7d7cda (0xe4b9 bytes) 2024-04-23T21:22:29.331 INFO:teuthology.orchestra.run.smithi111.stdout:5548 read 0x5e566d thru 0x5ef32b (0x9cbf bytes) 2024-04-23T21:22:29.331 INFO:teuthology.orchestra.run.smithi111.stdout:5549 read 0x37d37b thru 0x38b041 (0xdcc7 bytes) 2024-04-23T21:22:29.331 INFO:teuthology.orchestra.run.smithi111.stdout:5551 read 0x1cc1ff1 thru 0x1ccd874 (0xb884 bytes) 2024-04-23T21:22:29.332 INFO:teuthology.orchestra.run.smithi111.stdout:5552 read 0x3297b03 thru 0x32a3a4e (0xbf4c bytes) 2024-04-23T21:22:29.332 INFO:teuthology.orchestra.run.smithi111.stdout:5554 read 0x61c3cd thru 0x625465 (0x9099 bytes) 2024-04-23T21:22:29.346 INFO:teuthology.orchestra.run.smithi111.stdout:5555 write 0x9dcd027 thru 0x9dcd052 (0x2c bytes) 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:131315 3.a 3:54b189fc:::rbd_data.1118ba99ca9c.0000000000000007:head [set-alloc-hint object_size 16777216 write_size 16777216,write 11325479~44 in=44b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:29.350358+0000", 2024-04-23T21:22:29.350 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0011222680000000001, 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011426800000000001, 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 131315 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350358+0000", 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350358+0000", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350360+0000", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1959999999999998e-06 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350370+0000", 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7540000000000001e-06 2024-04-23T21:22:29.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350392+0000", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1705e-05 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350402+0000", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0596e-05 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350461+0000", 2024-04-23T21:22:29.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8934999999999997e-05 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350519+0000", 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7927999999999998e-05 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.350610+0000", 2024-04-23T21:22:29.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0859000000000005e-05 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:29.351338+0000", 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00072829600000000002 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:22:29.355 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:29.356 INFO:teuthology.orchestra.run.smithi111.stdout:5556 punch from 0x8c65d96 to 0x8c7447c, (0xe6e6 bytes) 2024-04-23T21:22:29.356 INFO:teuthology.orchestra.run.smithi111.stderr:marked out osd.3. 2024-04-23T21:22:29.357 INFO:teuthology.orchestra.run.smithi111.stdout:5558 read 0x431cc21 thru 0x4322f05 (0x62e5 bytes) 2024-04-23T21:22:29.357 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:22:29.357 INFO:teuthology.orchestra.run.smithi111.stdout:5561 writesame 0x758f44d thru 0x759902f data_size 0x1(0x9be3 bytes) 2024-04-23T21:22:29.361 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:29.407 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:29.407+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:29.499 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:29.500+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:29.504 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:29.504 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:29.504 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:29.504 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:29.504 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:29.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:29.601 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:29.602+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004387334999997, 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.659 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:29.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004366210999997, 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:29.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004336944000002, 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:29.672 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004316181999997, 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:29.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:29.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004297156, 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:29.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:29.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.680 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004278317000001, 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:29.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:29.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:29.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:29.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004244915000001, 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004225169000001, 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:29.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004207065999999, 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.694 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:29.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:29.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:29.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004187864999999, 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:29.698 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:29.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004162465999997, 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:29.706 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004137341000003, 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:29.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004112329999998, 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:29.711 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.715 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004088760999998, 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004065605999998, 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:29.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:29.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:29.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004044862999997, 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:29.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004026562, 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.004008982000002, 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:29.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 38.003979459, 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:29.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:29.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:29.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.264724395000002, 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:29.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:29.746 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:29.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:29.748 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:29.748+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:29.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:29.848 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:29.848+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:29.893 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:29.893 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:29.893 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:29.893 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:29.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:29.918 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:29.918 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:29.934 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:29.934 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:29.949 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:29.950+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:29.951 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:29.951 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:29.967 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:29.967 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:29.985 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:29.986 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:30.002 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:30.003 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:30.005 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:30.005 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:30.005 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:30.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:30.019 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:30.020 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:30.037 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:30.037 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:30.089 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:30.089+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:30.151 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:30.152+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:30.156 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.156 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:30.252 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:30.252+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:30.268 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.268 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:30.354 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:30.354+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:30.423 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.423 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.432 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:30.476 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:30.476+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:30.534 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.534 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:30.584 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:30.584+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:30.656 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:30.656+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:30.686 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.686 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.695 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:30.757 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:30.758+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.837 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.845 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:30.857 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:30.858+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:30.948 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:30.948 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:30.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:30.958 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:30.960+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:31.059 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.060 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.060 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:31.060+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:31.068 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:31.161 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:31.162+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:31.210 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.210 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.218 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:31.262 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:31.262+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:31.319 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.320 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:31.363 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:31.364+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:31.366 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:22:31.410 INFO:teuthology.orchestra.run.smithi111.stdout:5562 trunc from 0x9dcd053 to 0x9102aea 2024-04-23T21:22:31.441 INFO:teuthology.orchestra.run.smithi111.stdout:5563 write 0xed3384c thru 0xed3fed2 (0xc687 bytes) 2024-04-23T21:22:31.445 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.445 INFO:teuthology.orchestra.run.smithi111.stdout:5567 read 0x9d804c thru 0x9d86bd (0x672 bytes) 2024-04-23T21:22:31.445 INFO:teuthology.orchestra.run.smithi111.stdout:5570 write 0xd7c1585 thru 0xd7c580d (0x4289 bytes) 2024-04-23T21:22:31.450 INFO:teuthology.orchestra.run.smithi111.stdout:5571 writesame 0xc4d9865 thru 0xc4e801c data_size 0x14(0xe7b8 bytes) 2024-04-23T21:22:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:5574 read 0xd51afe9 thru 0xd526053 (0xb06b bytes) 2024-04-23T21:22:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:5575 punch from 0xe24694d to 0xe2535d1, (0xcc84 bytes) 2024-04-23T21:22:31.458 INFO:teuthology.orchestra.run.smithi111.stdout:5578 writesame 0xa64975a thru 0xa659527 data_size 0xb6(0xfdce bytes) 2024-04-23T21:22:31.463 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:31.464+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:31.466 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.466 INFO:teuthology.orchestra.run.smithi111.stdout:5581 write 0x2a3f581 thru 0x2a41625 (0x20a5 bytes) 2024-04-23T21:22:31.468 INFO:teuthology.orchestra.run.smithi111.stdout:5582 write 0xe71d2f4 thru 0xe723aaf (0x67bc bytes) 2024-04-23T21:22:31.478 INFO:teuthology.orchestra.run.smithi111.stdout:5583 trunc from 0xed3fed3 to 0x88e5e4d 2024-04-23T21:22:31.498 INFO:teuthology.orchestra.run.smithi111.stdout:5584 punch from 0x228c316 to 0x229a2f0, (0xdfda bytes) 2024-04-23T21:22:31.511 INFO:teuthology.orchestra.run.smithi111.stdout:5585 write 0xa85d47a thru 0xa869778 (0xc2ff bytes) 2024-04-23T21:22:31.512 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.512 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.514 INFO:teuthology.orchestra.run.smithi111.stdout:5586 read 0x6afa314 thru 0x6aff785 (0x5472 bytes) 2024-04-23T21:22:31.515 INFO:teuthology.orchestra.run.smithi111.stdout:5587 write 0x9b44876 thru 0x9b4bb3e (0x72c9 bytes) 2024-04-23T21:22:31.520 INFO:teuthology.orchestra.run.smithi111.stdout:5588 writesame 0x80dd83b thru 0x80de0b9 data_size 0x19(0x87f bytes) 2024-04-23T21:22:31.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:31.529 INFO:teuthology.orchestra.run.smithi111.stdout:5590 trunc from 0xa869779 to 0xc6763ed 2024-04-23T21:22:31.538 INFO:teuthology.orchestra.run.smithi111.stdout:5591 trunc from 0xc6763ed to 0xe191e6f 2024-04-23T21:22:31.560 INFO:teuthology.orchestra.run.smithi111.stdout:5593 trunc from 0xe191e6f to 0x4266831 2024-04-23T21:22:31.564 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:31.565+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:31.587 INFO:teuthology.orchestra.run.smithi111.stdout:5594 punch from 0x17a037a to 0x17a308c, (0x2d12 bytes) 2024-04-23T21:22:31.587 INFO:teuthology.orchestra.run.smithi111.stdout:5598 read 0x4143b43 thru 0x414ed66 (0xb224 bytes) 2024-04-23T21:22:31.588 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.602 INFO:teuthology.orchestra.run.smithi111.stdout:5600 trunc from 0x4266831 to 0xd8f59f5 2024-04-23T21:22:31.610 INFO:teuthology.orchestra.run.smithi111.stdout:5601 writesame 0x464dc1e thru 0x4658cbc data_size 0x1(0xb09f bytes) 2024-04-23T21:22:31.634 INFO:teuthology.orchestra.run.smithi111.stdout:5602 write 0xcee31bd thru 0xcee43ad (0x11f1 bytes) 2024-04-23T21:22:31.640 INFO:teuthology.orchestra.run.smithi111.stdout:5603 write 0x46f9b42 thru 0x4709248 (0xf707 bytes) 2024-04-23T21:22:31.645 INFO:teuthology.orchestra.run.smithi111.stdout:5604 read 0x5c8314b thru 0x5c8a635 (0x74eb bytes) 2024-04-23T21:22:31.645 INFO:teuthology.orchestra.run.smithi111.stdout:5606 write 0x8c1950f thru 0x8c1e3aa (0x4e9c bytes) 2024-04-23T21:22:31.665 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:31.665+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:31.668 INFO:teuthology.orchestra.run.smithi111.stdout:5608 trunc from 0xd8f59f5 to 0x35b8e25 2024-04-23T21:22:31.681 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.681 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:31.693 INFO:teuthology.orchestra.run.smithi111.stdout:5609 read 0x1026d12 thru 0x1034f3f (0xe22e bytes) 2024-04-23T21:22:31.705 INFO:teuthology.orchestra.run.smithi111.stdout:5611 writesame 0x62ee182 thru 0x62f1215 data_size 0x2(0x3094 bytes) 2024-04-23T21:22:31.705 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.705 INFO:teuthology.orchestra.run.smithi111.stdout:5614 write 0x3f3e44c thru 0x3f49fa7 (0xbb5c bytes) 2024-04-23T21:22:31.707 INFO:teuthology.orchestra.run.smithi111.stdout:5615 read 0x5b0f2b4 thru 0x5b16d12 (0x7a5f bytes) 2024-04-23T21:22:31.715 INFO:teuthology.orchestra.run.smithi111.stdout:5616 trunc from 0x62f1216 to 0xcba3978 2024-04-23T21:22:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:5617 read 0x68e37bc thru 0x68f2880 (0xf0c5 bytes) 2024-04-23T21:22:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:5621 punch from 0xaa8ebf1 to 0xaa98bdb, (0x9fea bytes) 2024-04-23T21:22:31.722 INFO:teuthology.orchestra.run.smithi111.stdout:5622 read 0x7f8736a thru 0x7f8d132 (0x5dc9 bytes) 2024-04-23T21:22:31.723 INFO:teuthology.orchestra.run.smithi111.stdout:5623 write 0x9b63ab4 thru 0x9b66f7d (0x34ca bytes) 2024-04-23T21:22:31.726 INFO:teuthology.orchestra.run.smithi111.stdout:5624 writesame 0x326d610 thru 0x3275f1f data_size 0x33(0x8910 bytes) 2024-04-23T21:22:31.728 INFO:teuthology.orchestra.run.smithi111.stdout:5625 read 0xada8a25 thru 0xada9faf (0x158b bytes) 2024-04-23T21:22:31.730 INFO:teuthology.orchestra.run.smithi111.stdout:5626 trunc from 0xcba3978 to 0xbba9f5a 2024-04-23T21:22:31.738 INFO:teuthology.orchestra.run.smithi111.stdout:5627 write 0x8501155 thru 0x8507577 (0x6423 bytes) 2024-04-23T21:22:31.746 INFO:teuthology.orchestra.run.smithi111.stdout:5628 trunc from 0xbba9f5a to 0xa669d2a 2024-04-23T21:22:31.756 INFO:teuthology.orchestra.run.smithi111.stdout:5629 read 0x19baaaf thru 0x19be78d (0x3cdf bytes) 2024-04-23T21:22:31.761 INFO:teuthology.orchestra.run.smithi111.stdout:5631 trunc from 0xa669d2a to 0xe8af30d 2024-04-23T21:22:31.766 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:31.767+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:31.768 INFO:teuthology.orchestra.run.smithi111.stdout:5633 writesame 0x6b66458 thru 0x6b6defd data_size 0x3(0x7aa6 bytes) 2024-04-23T21:22:31.775 INFO:teuthology.orchestra.run.smithi111.stdout:5634 read 0xb4a1e41 thru 0xb4a8eab (0x706b bytes) 2024-04-23T21:22:31.775 INFO:teuthology.orchestra.run.smithi111.stdout:5635 read 0x51ec02c thru 0x51f2ac9 (0x6a9e bytes) 2024-04-23T21:22:31.776 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.776 INFO:teuthology.orchestra.run.smithi111.stdout:5639 write 0x4f10cd6 thru 0x4f18f72 (0x829d bytes) 2024-04-23T21:22:31.777 INFO:teuthology.orchestra.run.smithi111.stdout:5643 read 0xa787982 thru 0xa7892b9 (0x1938 bytes) 2024-04-23T21:22:31.778 INFO:teuthology.orchestra.run.smithi111.stdout:5644 punch from 0xdaa3cad to 0xdaa6972, (0x2cc5 bytes) 2024-04-23T21:22:31.778 INFO:teuthology.orchestra.run.smithi111.stdout:5645 punch from 0xcd3fe0f to 0xcd44750, (0x4941 bytes) 2024-04-23T21:22:31.779 INFO:teuthology.orchestra.run.smithi111.stdout:5647 punch from 0x76ccc56 to 0x76d5b0c, (0x8eb6 bytes) 2024-04-23T21:22:31.779 INFO:teuthology.orchestra.run.smithi111.stdout:5648 read 0x408ee5e thru 0x40933b1 (0x4554 bytes) 2024-04-23T21:22:31.779 INFO:teuthology.orchestra.run.smithi111.stdout:5650 punch from 0x65da144 to 0x65e8f93, (0xee4f bytes) 2024-04-23T21:22:31.779 INFO:teuthology.orchestra.run.smithi111.stdout:5653 writesame 0x9c66b37 thru 0x9c6c6d3 data_size 0x1(0x5b9d bytes) 2024-04-23T21:22:31.791 INFO:teuthology.orchestra.run.smithi111.stdout:5655 writesame 0x5876331 thru 0x5883d53 data_size 0x1(0xda23 bytes) 2024-04-23T21:22:31.814 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.814 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.818 INFO:teuthology.orchestra.run.smithi111.stdout:5656 read 0x45f6c9f thru 0x45fcbc8 (0x5f2a bytes) 2024-04-23T21:22:31.819 INFO:teuthology.orchestra.run.smithi111.stdout:5657 read 0xe2c0bf5 thru 0xe2caa81 (0x9e8d bytes) 2024-04-23T21:22:31.819 INFO:teuthology.orchestra.run.smithi111.stdout:5658 write 0x61522d0 thru 0x6155c92 (0x39c3 bytes) 2024-04-23T21:22:31.824 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:31.835 INFO:teuthology.orchestra.run.smithi111.stdout:5659 trunc from 0xe8af30d to 0x6aa06b0 2024-04-23T21:22:31.853 INFO:teuthology.orchestra.run.smithi111.stdout:5660 trunc from 0x6aa06b0 to 0xdda9aca 2024-04-23T21:22:31.860 INFO:teuthology.orchestra.run.smithi111.stdout:5661 punch from 0x82fa9c9 to 0x8304cfa, (0xa331 bytes) 2024-04-23T21:22:31.864 INFO:teuthology.orchestra.run.smithi111.stdout:5665 trunc from 0xdda9aca to 0xa729aeb 2024-04-23T21:22:31.867 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:31.867+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:31.886 INFO:teuthology.orchestra.run.smithi111.stdout:5666 write 0xdb1142b thru 0xdb129f4 (0x15ca bytes) 2024-04-23T21:22:31.889 INFO:teuthology.orchestra.run.smithi111.stdout:5667 write 0x45688a6 thru 0x456a659 (0x1db4 bytes) 2024-04-23T21:22:31.899 INFO:teuthology.orchestra.run.smithi111.stdout:5668 trunc from 0xdb129f5 to 0x6e75f6e 2024-04-23T21:22:31.922 INFO:teuthology.orchestra.run.smithi111.stdout:5669 write 0xaacc9ef thru 0xaad984a (0xce5c bytes) 2024-04-23T21:22:31.934 INFO:teuthology.orchestra.run.smithi111.stdout:5671 trunc from 0xaad984b to 0x4f2f817 2024-04-23T21:22:31.948 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:31.948 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:31.953 INFO:teuthology.orchestra.run.smithi111.stdout:5672 writesame 0x45b2c6c thru 0x45b5bb5 data_size 0x1(0x2f4a bytes) 2024-04-23T21:22:31.958 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:31.960 INFO:teuthology.orchestra.run.smithi111.stdout:5673 read 0x77017e thru 0x776829 (0x66ac bytes) 2024-04-23T21:22:31.960 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.960 INFO:teuthology.orchestra.run.smithi111.stdout:5675 read 0x962129 thru 0x96dae7 (0xb9bf bytes) 2024-04-23T21:22:31.960 INFO:teuthology.orchestra.run.smithi111.stdout:5676 read 0x148a66f thru 0x148e55f (0x3ef1 bytes) 2024-04-23T21:22:31.968 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:31.968+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:31.971 INFO:teuthology.orchestra.run.smithi111.stdout:5680 writesame 0x7774b87 thru 0x7781ed7 data_size 0x1(0xd351 bytes) 2024-04-23T21:22:31.971 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:31.983 INFO:teuthology.orchestra.run.smithi111.stdout:5682 write 0xba8d49d thru 0xba9d266 (0xfdca bytes) 2024-04-23T21:22:31.986 INFO:teuthology.orchestra.run.smithi111.stdout:5683 punch from 0x2a95b5f to 0x2aa1b58, (0xbff9 bytes) 2024-04-23T21:22:31.987 INFO:teuthology.orchestra.run.smithi111.stdout:5685 writesame 0x48717b3 thru 0x487c873 data_size 0x1(0xb0c1 bytes) 2024-04-23T21:22:32.008 INFO:teuthology.orchestra.run.smithi111.stdout:5686 writesame 0x4010dd3 thru 0x401c2a8 data_size 0x2(0xb4d6 bytes) 2024-04-23T21:22:32.018 INFO:teuthology.orchestra.run.smithi111.stdout:5687 write 0x829a8e2 thru 0x829eb32 (0x4251 bytes) 2024-04-23T21:22:32.022 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:32.022 INFO:teuthology.orchestra.run.smithi111.stdout:5689 read 0x830556 thru 0x8399ac (0x9457 bytes) 2024-04-23T21:22:32.022 INFO:teuthology.orchestra.run.smithi111.stdout:5690 write 0x26dc929 thru 0x26dd9fb (0x10d3 bytes) 2024-04-23T21:22:32.024 INFO:teuthology.orchestra.run.smithi111.stdout:5691 read 0x49ca5d5 thru 0x49d1459 (0x6e85 bytes) 2024-04-23T21:22:32.024 INFO:teuthology.orchestra.run.smithi111.stdout:5692 writesame 0x6abd472 thru 0x6accbe0 data_size 0x1(0xf76f bytes) 2024-04-23T21:22:32.049 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:32.055 INFO:teuthology.orchestra.run.smithi111.stdout:5693 read 0x7aec591 thru 0x7aeed41 (0x27b1 bytes) 2024-04-23T21:22:32.055 INFO:teuthology.orchestra.run.smithi111.stdout:5694 write 0x8692b8d thru 0x86a0a94 (0xdf08 bytes) 2024-04-23T21:22:32.063 INFO:teuthology.orchestra.run.smithi111.stdout:5695 trunc from 0xba9d267 to 0x8351ff9 2024-04-23T21:22:32.069 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:32.069+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:32.070 INFO:teuthology.orchestra.run.smithi111.stdout:5696 write 0x125e39a thru 0x1268911 (0xa578 bytes) 2024-04-23T21:22:32.072 INFO:teuthology.orchestra.run.smithi111.stdout:5697 read 0x571767a thru 0x5723641 (0xbfc8 bytes) 2024-04-23T21:22:32.073 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:32.073 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:32.073 INFO:teuthology.orchestra.run.smithi111.stdout:5699 read 0x7ba1fba thru 0x7bb130a (0xf351 bytes) 2024-04-23T21:22:32.077 INFO:teuthology.orchestra.run.smithi111.stdout:5700 trunc from 0x8351ff9 to 0x4344624 2024-04-23T21:22:32.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:32.102 INFO:teuthology.orchestra.run.smithi111.stdout:truncating to largest ever: 0xee4d6b2 2024-04-23T21:22:32.117 INFO:teuthology.orchestra.run.smithi111.stdout:5702 trunc from 0x4344624 to 0xee4d6b2 2024-04-23T21:22:32.127 INFO:teuthology.orchestra.run.smithi111.stdout:5703 trunc from 0xee4d6b2 to 0xbcd7346 2024-04-23T21:22:32.137 INFO:teuthology.orchestra.run.smithi111.stdout:5704 trunc from 0xbcd7346 to 0xe9d1acb 2024-04-23T21:22:32.144 INFO:teuthology.orchestra.run.smithi111.stdout:5705 writesame 0x539be08 thru 0x53a3f1c data_size 0x5(0x8115 bytes) 2024-04-23T21:22:32.148 INFO:teuthology.orchestra.run.smithi111.stdout:5706 write 0x215e7e4 thru 0x21680e0 (0x98fd bytes) 2024-04-23T21:22:32.161 INFO:teuthology.orchestra.run.smithi111.stdout:5709 trunc from 0xe9d1acb to 0x5464216 2024-04-23T21:22:32.170 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:32.170+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:32.174 INFO:teuthology.orchestra.run.smithi111.stdout:5711 read 0xb7c269 thru 0xb83501 (0x7299 bytes) 2024-04-23T21:22:32.174 INFO:teuthology.orchestra.run.smithi111.stdout:5712 punch from 0x6e53c9 to 0x6e94b6, (0x40ed bytes) 2024-04-23T21:22:32.174 INFO:teuthology.orchestra.run.smithi111.stdout:5713 read 0x485479f thru 0x485f8ce (0xb130 bytes) 2024-04-23T21:22:32.175 INFO:teuthology.orchestra.run.smithi111.stdout:5714 punch from 0x29695f0 to 0x296bef6, (0x2906 bytes) 2024-04-23T21:22:32.175 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:32.175 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:32.179 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:22:32.182 INFO:teuthology.orchestra.run.smithi111.stdout:5717 trunc from 0x5464216 to 0x9c735f 2024-04-23T21:22:32.199 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:32.200 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:32.203 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:32.203 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:32.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:32.222 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:32.222 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:32.241 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:32.242 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:32.260 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:32.261 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:32.271 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:32.271+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:32.280 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:32.280 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:32.297 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:32.298 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:32.315 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:32.316 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:32.319 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:32.320 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:32.320 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:32.320 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:32.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:32.333 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:32.333 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:32.385 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:32.385+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:32.433 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:32.433 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:32.433 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:32.433 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:32.433 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:32.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:32.477 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:32.477+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:32.546 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:32.546 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:32.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.670376908, 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.547 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:32.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:32.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:32.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.603958399, 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.551 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:32.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.603941018, 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.555 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:32.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.559 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.603923461000001, 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.602263266, 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.567 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.60224137, 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:32.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:32.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:32.571 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.60221402, 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.572 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.579 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.602189932, 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.580 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.602158873, 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.584 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.600958804999999, 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.600936372, 2024-04-23T21:22:32.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:32.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:32.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.596 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.600912784, 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.600892845000001, 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:32.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.604 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.60086437, 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:32.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.608 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.599481625999999, 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.609 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.594385001999999, 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.613 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:32.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:32.617 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.594038376, 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:32.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.582523472, 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.582503533000001, 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:32.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 12.581203262000001, 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:32.630 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:32.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:32.634 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:32.635 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:32.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:32.637 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:32.636+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:32.737 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:32.738+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:32.779 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:32.779 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:32.779 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:32.779 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:32.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:32.837 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:32.838+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:32.889 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:32.889 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:32.889 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:32.889 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:32.889 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:32.898 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:32.938 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:32.939+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.002 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347970607000001, 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347949483000001, 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:33.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347920215999999, 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:33.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:33.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.015 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347899454, 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:33.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347880428000003, 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347861588999997, 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:33.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.027 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347828186999998, 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:33.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:33.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347808440999998, 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:33.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347790338000003, 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:33.036 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:33.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:33.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347771137000002, 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:33.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347745738, 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.045 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:33.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:33.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347720613, 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.049 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:33.053 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347695602000002, 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347672033000002, 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:33.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.061 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347648878000001, 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:33.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:33.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.065 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347628135000001, 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:33.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:33.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347609833999996, 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:33.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:33.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347592253999998, 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:33.074 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:33.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:33.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.347562730999996, 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.078 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:33.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:33.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.608307666999998, 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:33.082 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.087 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.088 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.088 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:33.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:33.090 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:33.090+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:33.191 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:33.192+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop(client.4376.0:131745 3.16 e89/18 3:6b2d743f:::rbd_header.1118ba99ca9c:head v 89'363, mlcod=87'361)", 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:33.198242+0000", 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.001164076, 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0011841320000000001, 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.198 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198242+0000", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198242+0000", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198242+0000", 2024-04-23T21:22:33.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.4700000000000001e-07 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198245+0000", 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6529999999999998e-06 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198256+0000", 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0818e-05 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198264+0000", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4179999999999997e-06 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198312+0000", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7901000000000001e-05 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:33.198331+0000", 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8949000000000001e-05 2024-04-23T21:22:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:22:33.202 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:33.207 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:33.293 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:33.294+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:33.309 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:33.309 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:33.309 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:33.309 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:33.309 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:33.318 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:33.395 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:33.395+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:33.422 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 28.295080574, 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.423 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.424 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:33.425 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.426 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:33.427 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:33.428 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.495106829999999, 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:22:33.429 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.430 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.495084171, 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.434 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:33.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.438 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.495057651, 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.439 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.441 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.495037587000001, 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.443 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:33.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:22:33.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.447 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494744917, 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.448 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:22:33.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:22:33.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494735012, 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:22:33.453 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:22:33.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.457 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494432231999999, 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.458 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:33.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494395552, 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.463 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:33.467 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494373929, 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494353684, 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.494152367, 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:22:33.476 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493717695999999, 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493450871, 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.486 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:22:33.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.49342867, 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:33.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:22:33.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:22:33.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:22:33.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493409080999999, 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493070768999999, 2024-04-23T21:22:33.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:22:33.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.503 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493050187, 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.504 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:22:33.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.493031162999999, 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:22:33.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:22:33.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.512 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 14.492804999000001, 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.513 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:22:33.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:33.517 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:33.519 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:33.519 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:33.519+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:33.619 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:33.620+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:33.626 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:33.626 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3.35861339 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.427797396, 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.42783258499999999, 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:33.627 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "waiting for peered", 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.628 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:33.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:33.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:33.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:33.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:22:33.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:22:33.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:33.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:33.956 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:33.957 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:33.957 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:33.957 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 1 2024-04-23T21:22:33.957 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:33.958 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:33.959 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:33.959+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:34.060 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:34.061+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:34.101 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:34.101 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:34.101 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:34.101 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:34.101 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:34.110 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:34.161 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:34.160+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:34.191 INFO:teuthology.orchestra.run.smithi111.stdout:5718 read 0x7d69d2 thru 0x7e37d4 (0xce03 bytes) 2024-04-23T21:22:34.191 INFO:teuthology.orchestra.run.smithi111.stdout:5720 trunc from 0x9c735f to 0xc1f7c4 2024-04-23T21:22:34.197 INFO:teuthology.orchestra.run.smithi111.stdout:5722 punch from 0x39fff2 to 0x3a7dc8, (0x7dd6 bytes) 2024-04-23T21:22:34.197 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:22:34.201 INFO:teuthology.orchestra.run.smithi111.stdout:5725 trunc from 0xc1f7c4 to 0x3643bcf 2024-04-23T21:22:34.206 INFO:teuthology.orchestra.run.smithi111.stdout:5726 writesame 0x2d7f55a thru 0x2d8a117 data_size 0x59(0xabbe bytes) 2024-04-23T21:22:34.209 INFO:teuthology.orchestra.run.smithi111.stdout:5727 read 0x13b04e0 thru 0x13b5a10 (0x5531 bytes) 2024-04-23T21:22:34.215 INFO:teuthology.orchestra.run.smithi111.stdout:5729 trunc from 0x3643bcf to 0x8506104 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.220 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 45.279110875000001, 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:34.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:34.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:34.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70924 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 2621440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770153+0000", 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449742530000002, 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087492497000000002, 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70924 2024-04-23T21:22:34.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770153+0000", 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770153+0000", 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770154+0000", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770156+0000", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770168+0000", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1744e-05 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770177+0000", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4709999999999996e-06 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857557+0000", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087379643000000007 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857598+0000", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1341999999999998e-05 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857646+0000", 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7527000000000001e-05 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449711946000001, 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:22:34.230 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:22:34.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449689012, 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.235 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:34.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:22:34.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:22:34.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.240 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449663693000002, 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449644974000002, 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:22:34.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:22:34.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:22:34.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449602293000002, 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.250 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:34.254 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449576152999999, 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:34.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:22:34.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:34.259 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449558235000001, 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:22:34.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.263 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449526855999999, 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.264 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:22:34.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.269 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449509726999999, 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:22:34.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449481189, 2024-04-23T21:22:34.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449438204, 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:22:34.281 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.282 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:34.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:34.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.285 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.285 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:34.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:22:34.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:22:34.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:22:34.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.289 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:22:34.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.291 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449410966999999, 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.292 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:22:34.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449387179999999, 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.300 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:22:34.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:22:34.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449361937999999, 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:22:34.309 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:34.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:22:34.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.315 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449343171999999, 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:22:34.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.318 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.319 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449286212000001, 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:22:34.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:34.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:22:34.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:22:34.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449264478, 2024-04-23T21:22:34.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.327 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:22:34.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 20.449240257, 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:22:34.331 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:34.335 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5731 writesame 0x1de99f0 thru 0x1deca74 data_size 0x1(0x3085 bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5732 read 0x43d2a0d thru 0x43dbb90 (0x9184 bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5734 read 0x1bf80dc thru 0x1bfeebc (0x6de1 bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5735 punch from 0x5f238f9 to 0x5f30985, (0xd08c bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5737 punch from 0x378c35c to 0x379b56c, (0xf210 bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5738 punch from 0x6c3268a to 0x6c34af0, (0x2466 bytes) 2024-04-23T21:22:34.337 INFO:teuthology.orchestra.run.smithi111.stdout:5739 read 0x80c2159 thru 0x80c88ed (0x6795 bytes) 2024-04-23T21:22:34.339 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5742 trunc from 0x8506104 to 0xcbccbf4 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5743 write 0xc22572f thru 0xc22e9c8 (0x929a bytes) 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5744 writesame 0xbfdefb2 thru 0xbfe40d6 data_size 0x1(0x5125 bytes) 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5745 trunc from 0xcbccbf4 to 0xda66793 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5746 trunc from 0xda66793 to 0x10035ab 2024-04-23T21:22:34.340 INFO:teuthology.orchestra.run.smithi111.stdout:5747 write 0x5d93cb3 thru 0x5d978f5 (0x3c43 bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5748 write 0x30431ae thru 0x304dacd (0xa920 bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5749 writesame 0xf4f50a thru 0xf5879f data_size 0x1d(0x9296 bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5750 punch from 0x45dc221 to 0x45e3383, (0x7162 bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5751 read 0x17be74 thru 0x17e78e (0x291b bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5753 trunc from 0x5d978f6 to 0xb37a996 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5754 write 0x8df99d8 thru 0x8dff5c6 (0x5bef bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5759 read 0x40645b thru 0x40a651 (0x41f7 bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5760 read 0x9ff9af0 thru 0xa007eac (0xe3bd bytes) 2024-04-23T21:22:34.341 INFO:teuthology.orchestra.run.smithi111.stdout:5761 writesame 0x207e349 thru 0x2087f81 data_size 0x1(0x9c39 bytes) 2024-04-23T21:22:34.341 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:34.337+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:34.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:34.352 INFO:teuthology.orchestra.run.smithi111.stdout:5764 trunc from 0xb37a996 to 0x885358 2024-04-23T21:22:34.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:34.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-23T21:22:34.370 INFO:teuthology.orchestra.run.smithi111.stdout:5765 read 0x5179f8 thru 0x520f84 (0x958d bytes) 2024-04-23T21:22:34.372 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.396 INFO:teuthology.orchestra.run.smithi111.stdout:5769 write 0xcca5f1a thru 0xccb5048 (0xf12f bytes) 2024-04-23T21:22:34.406 INFO:teuthology.orchestra.run.smithi111.stdout:5771 trunc from 0xccb5049 to 0x82650b5 2024-04-23T21:22:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:5772 punch from 0x11bd4aa to 0x11c92cf, (0xbe25 bytes) 2024-04-23T21:22:34.419 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.427 INFO:teuthology.orchestra.run.smithi111.stdout:5774 trunc from 0x82650b5 to 0xdee4ba2 2024-04-23T21:22:34.433 INFO:teuthology.orchestra.run.smithi111.stdout:5775 read 0x9301724 thru 0x9308147 (0x6a24 bytes) 2024-04-23T21:22:34.433 INFO:teuthology.orchestra.run.smithi111.stdout:5776 write 0x43c6e2a thru 0x43d64a5 (0xf67c bytes) 2024-04-23T21:22:34.435 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.435 INFO:teuthology.orchestra.run.smithi111.stdout:5778 punch from 0x7fe2d8d to 0x7feb357, (0x85ca bytes) 2024-04-23T21:22:34.437 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:34.438+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:34.441 INFO:teuthology.orchestra.run.smithi111.stdout:5779 trunc from 0xdee4ba2 to 0x92c87a0 2024-04-23T21:22:34.447 INFO:teuthology.orchestra.run.smithi111.stdout:5782 write 0x213ac5e thru 0x213dee2 (0x3285 bytes) 2024-04-23T21:22:34.450 INFO:teuthology.orchestra.run.smithi111.stdout:5783 write 0x706d1f1 thru 0x706e124 (0xf34 bytes) 2024-04-23T21:22:34.454 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:34.454 INFO:teuthology.orchestra.run.smithi111.stdout:5785 write 0x87e5b23 thru 0x87f406e (0xe54c bytes) 2024-04-23T21:22:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:5786 read 0x3dd577f thru 0x3dded87 (0x9609 bytes) 2024-04-23T21:22:34.458 INFO:teuthology.orchestra.run.smithi111.stdout:5787 write 0x6cb5055 thru 0x6cbea88 (0x9a34 bytes) 2024-04-23T21:22:34.462 INFO:teuthology.orchestra.run.smithi111.stdout:5789 writesame 0x40e2fa2 thru 0x40e3e6c data_size 0x1(0xecb bytes) 2024-04-23T21:22:34.474 INFO:teuthology.orchestra.run.smithi111.stdout:5790 write 0xacf83cc thru 0xad05433 (0xd068 bytes) 2024-04-23T21:22:34.477 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:34.477 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:34.477 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:34.477 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:34.478 INFO:teuthology.orchestra.run.smithi111.stdout:5791 read 0x35b9b42 thru 0x35bc7e3 (0x2ca2 bytes) 2024-04-23T21:22:34.478 INFO:teuthology.orchestra.run.smithi111.stdout:5792 write 0x3e679be thru 0x3e70cc2 (0x9305 bytes) 2024-04-23T21:22:34.479 INFO:teuthology.orchestra.run.smithi111.stdout:5794 clone 47 order 21 su 131072 sc 7 2024-04-23T21:22:34.487 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:34.494 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:34.494 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:34.519 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:34.520 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:34.538 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:34.538+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:34.554 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:34.555 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:34.572 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:34.572 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:34.600 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:34.600 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:34.605 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:34.605 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:34.609 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:34.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:34.629 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:34.639 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:34.640+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:34.646 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:34.646 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:34.663 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:34.664 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:34.732 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:34.734 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:34.739 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:34.740+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:34.741 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:34.841 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:34.842+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:34.884 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:34.884 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:34.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:34.942 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:34.942+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:34.995 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:34.995 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.003 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:35.043 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:35.044+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:35.144 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:35.144+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:35.146 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.147 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:35.247 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:35.247+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:35.299 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.299 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:35.348 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:35.349+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:35.369 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T21:22:35.372 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:22:35.385 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-23T21:22:35.419 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.419 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.429 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:35.441 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone46 from 0xad05434 (overlap 0x885358) to 0x0 2024-04-23T21:22:35.484 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:35.484+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:35.496 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #45, image image_client.0-clone45 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent46 2024-04-23T21:22:35.550 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:35.550+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.580 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:35.651 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:35.651+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:35.693 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.693 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:35.734 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:35.734 INFO:teuthology.orchestra.run.smithi111.stdout:5797 read 0x5bd5ed1 thru 0x5be1fba (0xc0ea bytes) 2024-04-23T21:22:35.735 INFO:teuthology.orchestra.run.smithi111.stdout:5798 read 0x87f5b32 thru 0x87ff8ae (0x9d7d bytes) 2024-04-23T21:22:35.736 INFO:teuthology.orchestra.run.smithi111.stdout:5799 write 0x576af1c thru 0x577861d (0xd702 bytes) 2024-04-23T21:22:35.745 INFO:teuthology.orchestra.run.smithi111.stdout:5800 write 0x8ccd2b9 thru 0x8cd8996 (0xb6de bytes) 2024-04-23T21:22:35.750 INFO:teuthology.orchestra.run.smithi111.stdout:5801 trunc from 0xad05434 to 0x9d8e18c 2024-04-23T21:22:35.753 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:35.752+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:35.773 INFO:teuthology.orchestra.run.smithi111.stdout:5802 write 0x56870f8 thru 0x5694088 (0xcf91 bytes) 2024-04-23T21:22:35.775 INFO:teuthology.orchestra.run.smithi111.stdout:5803 write 0x1b698e6 thru 0x1b784d0 (0xebeb bytes) 2024-04-23T21:22:35.779 INFO:teuthology.orchestra.run.smithi111.stdout:5804 clone 48 order 23 su 2097152 sc 9 2024-04-23T21:22:35.812 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.812 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.820 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:35.854 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:35.855+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:35.955 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:35.954+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:35.963 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:35.963 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:35.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:36.056 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:36.056+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:36.115 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.115 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.124 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:36.166 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:36.165+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:36.257 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:36.257+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:36.269 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.269 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.277 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:36.361 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:36.361+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:36.373 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T21:22:36.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-23T21:22:36.428 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.428 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.437 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:36.443 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone47 from 0x9d8e18c (overlap 0x9d8e18c) to 0x7e0b470 2024-04-23T21:22:36.479 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:36.478+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:36.559 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:36.560+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:36.575 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #46, image image_client.0-clone46 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent47 2024-04-23T21:22:36.576 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.586 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.586 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.591 INFO:teuthology.orchestra.run.smithi111.stdout:5806 write 0xe4c1dcb thru 0xe4c2682 (0x8b8 bytes) 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:5811 punch from 0x2c97737 to 0x2ca72fa, (0xfbc3 bytes) 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.595 INFO:teuthology.orchestra.run.smithi111.stdout:5814 write 0x26218df thru 0x2626939 (0x505b bytes) 2024-04-23T21:22:36.596 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:36.626 INFO:teuthology.orchestra.run.smithi111.stdout:5816 punch from 0x22246f7 to 0x22297e9, (0x50f2 bytes) 2024-04-23T21:22:36.626 INFO:teuthology.orchestra.run.smithi111.stdout:5817 read 0x9f607ad thru 0x9f6a715 (0x9f69 bytes) 2024-04-23T21:22:36.626 INFO:teuthology.orchestra.run.smithi111.stdout:5819 write 0x2e3f86c thru 0x2e4f3d2 (0xfb67 bytes) 2024-04-23T21:22:36.660 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:36.660+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:36.675 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:36.677 INFO:teuthology.orchestra.run.smithi111.stdout:5820 trunc from 0xe4c2683 to 0xc2b3c6 2024-04-23T21:22:36.720 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.720 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:36.730 INFO:teuthology.orchestra.run.smithi111.stdout:5821 write 0x5d09496 thru 0x5d12da0 (0x990b bytes) 2024-04-23T21:22:36.734 INFO:teuthology.orchestra.run.smithi111.stdout:5822 writesame 0x2ff0a7 thru 0x304185 data_size 0x43(0x50df bytes) 2024-04-23T21:22:36.737 INFO:teuthology.orchestra.run.smithi111.stdout:5823 write 0x1ba2c43 thru 0x1bb020c (0xd5ca bytes) 2024-04-23T21:22:36.739 INFO:teuthology.orchestra.run.smithi111.stdout:5826 write 0x539c726 thru 0x53a89ec (0xc2c7 bytes) 2024-04-23T21:22:36.743 INFO:teuthology.orchestra.run.smithi111.stdout:5828 punch from 0x5a9716d to 0x5a9d4ee, (0x6381 bytes) 2024-04-23T21:22:36.743 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.743 INFO:teuthology.orchestra.run.smithi111.stdout:5831 punch from 0x43351e4 to 0x4345039, (0xfe55 bytes) 2024-04-23T21:22:36.743 INFO:teuthology.orchestra.run.smithi111.stdout:5832 writesame 0x1353da0 thru 0x1363585 data_size 0x15(0xf7e6 bytes) 2024-04-23T21:22:36.745 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.745 INFO:teuthology.orchestra.run.smithi111.stdout:5834 punch from 0x4e7a822 to 0x4e7aa84, (0x262 bytes) 2024-04-23T21:22:36.745 INFO:teuthology.orchestra.run.smithi111.stdout:5836 read 0x3643453 thru 0x3651c50 (0xe7fe bytes) 2024-04-23T21:22:36.758 INFO:teuthology.orchestra.run.smithi111.stdout:5837 writesame 0xa1f2dc3 thru 0xa1f9ff1 data_size 0x1(0x722f bytes) 2024-04-23T21:22:36.758 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.758 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.758 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.761 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:36.761+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:36.768 INFO:teuthology.orchestra.run.smithi111.stdout:5843 write 0xd55e2f5 thru 0xd5663ec (0x80f8 bytes) 2024-04-23T21:22:36.773 INFO:teuthology.orchestra.run.smithi111.stdout:5844 write 0x171c380 thru 0x17249c5 (0x8646 bytes) 2024-04-23T21:22:36.775 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.775 INFO:teuthology.orchestra.run.smithi111.stdout:5846 read 0xbf20d5 thru 0xbf9059 (0x6f85 bytes) 2024-04-23T21:22:36.775 INFO:teuthology.orchestra.run.smithi111.stdout:5847 writesame 0x27d6195 thru 0x27dda18 data_size 0x12(0x7884 bytes) 2024-04-23T21:22:36.777 INFO:teuthology.orchestra.run.smithi111.stdout:5848 read 0x8f32f83 thru 0x8f3392f (0x9ad bytes) 2024-04-23T21:22:36.777 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.778 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.778 INFO:teuthology.orchestra.run.smithi111.stdout:5852 write 0x5b41e29 thru 0x5b45bbc (0x3d94 bytes) 2024-04-23T21:22:36.781 INFO:teuthology.orchestra.run.smithi111.stdout:5853 write 0x76a671e thru 0x76a8d4a (0x262d bytes) 2024-04-23T21:22:36.788 INFO:teuthology.orchestra.run.smithi111.stdout:5855 trunc from 0xd5663ed to 0x9c9c51e 2024-04-23T21:22:36.801 INFO:teuthology.orchestra.run.smithi111.stdout:5856 read 0x487d9e thru 0x489e71 (0x20d4 bytes) 2024-04-23T21:22:36.801 INFO:teuthology.orchestra.run.smithi111.stdout:5859 read 0xfb64e thru 0x107653 (0xc006 bytes) 2024-04-23T21:22:36.802 INFO:teuthology.orchestra.run.smithi111.stdout:5860 read 0x737e1b thru 0x73a589 (0x276f bytes) 2024-04-23T21:22:36.802 INFO:teuthology.orchestra.run.smithi111.stdout:5861 write 0x7dd8f8c thru 0x7de70e4 (0xe159 bytes) 2024-04-23T21:22:36.805 INFO:teuthology.orchestra.run.smithi111.stdout:5863 read 0x2fa5092 thru 0x2fa924c (0x41bb bytes) 2024-04-23T21:22:36.806 INFO:teuthology.orchestra.run.smithi111.stdout:5865 read 0x2e3720c thru 0x2e3de78 (0x6c6d bytes) 2024-04-23T21:22:36.813 INFO:teuthology.orchestra.run.smithi111.stdout:5866 write 0xb49c36b thru 0xb4a3fcd (0x7c63 bytes) 2024-04-23T21:22:36.817 INFO:teuthology.orchestra.run.smithi111.stdout:5867 read 0x56738f8 thru 0x568084f (0xcf58 bytes) 2024-04-23T21:22:36.818 INFO:teuthology.orchestra.run.smithi111.stdout:5868 read 0x8becf03 thru 0x8bef41b (0x2519 bytes) 2024-04-23T21:22:36.818 INFO:teuthology.orchestra.run.smithi111.stdout:5869 write 0x2f0d604 thru 0x2f1b935 (0xe332 bytes) 2024-04-23T21:22:36.820 INFO:teuthology.orchestra.run.smithi111.stdout:5870 read 0x6efabd3 thru 0x6f0573f (0xab6d bytes) 2024-04-23T21:22:36.820 INFO:teuthology.orchestra.run.smithi111.stdout:5871 read 0x17e1188 thru 0x17ef7fc (0xe675 bytes) 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:5874 read 0xab8d6b8 thru 0xab9b70c (0xe055 bytes) 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:5878 read 0xa674607 thru 0xa6810b3 (0xcaad bytes) 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:5879 read 0x543e37d thru 0x5444398 (0x601c bytes) 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:5885 writesame 0x90e0b23 thru 0x90ecd62 data_size 0xa8(0xc240 bytes) 2024-04-23T21:22:36.821 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:36.822 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:36.822 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.822 INFO:teuthology.orchestra.run.smithi111.stdout:5889 read 0x4b53b10 thru 0x4b55e59 (0x234a bytes) 2024-04-23T21:22:36.822 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.823 INFO:teuthology.orchestra.run.smithi111.stdout:5892 write 0xaed31a1 thru 0xaee0951 (0xd7b1 bytes) 2024-04-23T21:22:36.826 INFO:teuthology.orchestra.run.smithi111.stdout:5893 write 0x8b944b5 thru 0x8b9d328 (0x8e74 bytes) 2024-04-23T21:22:36.829 INFO:teuthology.orchestra.run.smithi111.stdout:5894 punch from 0x6750ae6 to 0x67593e9, (0x8903 bytes) 2024-04-23T21:22:36.838 INFO:teuthology.orchestra.run.smithi111.stdout:5896 write 0xeb8906a thru 0xeb8ed63 (0x5cfa bytes) 2024-04-23T21:22:36.840 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:36.841 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:36.841 INFO:teuthology.orchestra.run.smithi111.stdout:5897 writesame 0x16dbc23 thru 0x16e260f data_size 0x45(0x69ed bytes) 2024-04-23T21:22:36.843 INFO:teuthology.orchestra.run.smithi111.stdout:5898 write 0x8307be9 thru 0x83123d0 (0xa7e8 bytes) 2024-04-23T21:22:36.846 INFO:teuthology.orchestra.run.smithi111.stdout:5900 writesame 0x760ab3d thru 0x760f2e4 data_size 0x4(0x47a8 bytes) 2024-04-23T21:22:36.849 INFO:teuthology.orchestra.run.smithi111.stdout:5901 writesame 0x9a53210 thru 0x9a6246c data_size 0x1(0xf25d bytes) 2024-04-23T21:22:36.859 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:36.859 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:36.864 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:36.863+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:36.876 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:36.876 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:36.880 INFO:teuthology.orchestra.run.smithi111.stdout:5902 writesame 0xddd4c38 thru 0xddd92c7 data_size 0x8(0x4690 bytes) 2024-04-23T21:22:36.882 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:36.882 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:36.891 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:36.895 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:36.895 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:36.899 INFO:teuthology.orchestra.run.smithi111.stdout:5903 trunc from 0xeb8ed64 to 0x25bc866 2024-04-23T21:22:36.915 INFO:teuthology.orchestra.run.smithi111.stdout:5904 read 0xb05cc0 thru 0xb0ee44 (0x9185 bytes) 2024-04-23T21:22:36.915 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:36.915 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:36.927 INFO:teuthology.orchestra.run.smithi111.stdout:5905 write 0x7db84f8 thru 0x7dbb8f3 (0x33fc bytes) 2024-04-23T21:22:36.934 INFO:teuthology.orchestra.run.smithi111.stdout:5907 trunc from 0x7dbb8f4 to 0x536eec4 2024-04-23T21:22:36.936 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:36.936 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:36.943 INFO:teuthology.orchestra.run.smithi111.stdout:5908 read 0x14f9e83 thru 0x1504f7d (0xb0fb bytes) 2024-04-23T21:22:36.944 INFO:teuthology.orchestra.run.smithi111.stdout:5909 punch from 0x3f12a55 to 0x3f1a249, (0x77f4 bytes) 2024-04-23T21:22:36.950 INFO:teuthology.orchestra.run.smithi111.stdout:5911 trunc from 0x536eec4 to 0x43b02 2024-04-23T21:22:36.956 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:36.956 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:36.963 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:36.963+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:36.974 INFO:teuthology.orchestra.run.smithi111.stdout:5912 write 0xa848f0 thru 0xa9347a (0xeb8b bytes) 2024-04-23T21:22:36.978 INFO:teuthology.orchestra.run.smithi111.stdout:5914 punch from 0x34fa24 to 0x3548b9, (0x4e95 bytes) 2024-04-23T21:22:36.978 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:36.978 INFO:teuthology.orchestra.run.smithi111.stdout:5916 read 0x8958b5 thru 0x89cbe9 (0x7335 bytes) 2024-04-23T21:22:36.995 INFO:teuthology.orchestra.run.smithi111.stdout:5917 writesame 0xa4873c3 thru 0xa490b22 data_size 0x70(0x9760 bytes) 2024-04-23T21:22:36.995 INFO:teuthology.orchestra.run.smithi111.stdout:5918 write 0x31d15fa thru 0x31de8ee (0xd2f5 bytes) 2024-04-23T21:22:37.001 INFO:teuthology.orchestra.run.smithi111.stdout:5919 trunc from 0xa490b23 to 0x75e79d7 2024-04-23T21:22:37.004 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:37.004 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:133525 3.12 3:4b21a941:::rbd_object_map.1118a625dc11:head [call lock.assert_locked in=44b,call rbd.object_map_resize in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:37.004731+0000", 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.0014120510000000001, 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001430976, 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 133525 2024-04-23T21:22:37.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004731+0000", 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004731+0000", 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004732+0000", 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9699999999999995e-07 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004736+0000", 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8890000000000004e-06 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004746+0000", 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9669999999999996e-06 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004753+0000", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1489999999999997e-06 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004825+0000", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2259000000000001e-05 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.004913+0000", 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8041999999999995e-05 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 0", 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.005183+0000", 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00026960200000000003 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:37.006162+0000", 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00097927699999999997 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:22:37.009 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:37.010 INFO:teuthology.orchestra.run.smithi111.stdout:5920 trunc from 0x75e79d7 to 0xa4631d5 2024-04-23T21:22:37.014 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:37.016 INFO:teuthology.orchestra.run.smithi111.stdout:5921 write 0x62dd116 thru 0x62ecbee (0xfad9 bytes) 2024-04-23T21:22:37.021 INFO:teuthology.orchestra.run.smithi111.stdout:5922 read 0x252abdc thru 0x253427c (0x96a1 bytes) 2024-04-23T21:22:37.032 INFO:teuthology.orchestra.run.smithi111.stdout:5923 trunc from 0xa4631d5 to 0x13a2de3 2024-04-23T21:22:37.056 INFO:teuthology.orchestra.run.smithi111.stdout:5924 writesame 0x28d7107 thru 0x28db204 data_size 0x5e(0x40fe bytes) 2024-04-23T21:22:37.056 INFO:teuthology.orchestra.run.smithi111.stdout:5925 read 0x3e83e0 thru 0x3ea97e (0x259f bytes) 2024-04-23T21:22:37.056 INFO:teuthology.orchestra.run.smithi111.stdout:5926 write 0x1bcab25 thru 0x1bd41a0 (0x967c bytes) 2024-04-23T21:22:37.059 INFO:teuthology.orchestra.run.smithi111.stdout:5927 read 0x1ea5d0b thru 0x1ea7ce2 (0x1fd8 bytes) 2024-04-23T21:22:37.064 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:37.065+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:37.078 INFO:teuthology.orchestra.run.smithi111.stdout:5928 write 0xd045054 thru 0xd04ffee (0xaf9b bytes) 2024-04-23T21:22:37.082 INFO:teuthology.orchestra.run.smithi111.stdout:5929 read 0x98230cc thru 0x982fbe1 (0xcb16 bytes) 2024-04-23T21:22:37.082 INFO:teuthology.orchestra.run.smithi111.stdout:5931 write 0xa58f761 thru 0xa598c79 (0x9519 bytes) 2024-04-23T21:22:37.085 INFO:teuthology.orchestra.run.smithi111.stdout:5932 write 0x49119cc thru 0x4914614 (0x2c49 bytes) 2024-04-23T21:22:37.088 INFO:teuthology.orchestra.run.smithi111.stdout:5934 writesame 0x77e3e7c thru 0x77e8606 data_size 0x25(0x478b bytes) 2024-04-23T21:22:37.090 INFO:teuthology.orchestra.run.smithi111.stdout:5935 punch from 0x46d457 to 0x47c560, (0xf109 bytes) 2024-04-23T21:22:37.091 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.099 INFO:teuthology.orchestra.run.smithi111.stdout:5939 writesame 0xdefc2f9 thru 0xdf014ac data_size 0x54(0x51b4 bytes) 2024-04-23T21:22:37.099 INFO:teuthology.orchestra.run.smithi111.stdout:5940 writesame 0x5f473c6 thru 0x5f4e229 data_size 0x5a(0x6e64 bytes) 2024-04-23T21:22:37.103 INFO:teuthology.orchestra.run.smithi111.stdout:5941 read 0xf25ed4 thru 0xf2b8be (0x59eb bytes) 2024-04-23T21:22:37.104 INFO:teuthology.orchestra.run.smithi111.stdout:5942 read 0xcd18e5f thru 0xcd1bdd2 (0x2f74 bytes) 2024-04-23T21:22:37.114 INFO:teuthology.orchestra.run.smithi111.stdout:5943 trunc from 0xdf014ad to 0x53898b6 2024-04-23T21:22:37.127 INFO:teuthology.orchestra.run.smithi111.stdout:5944 write 0x49a3e4d thru 0x49b2234 (0xe3e8 bytes) 2024-04-23T21:22:37.131 INFO:teuthology.orchestra.run.smithi111.stdout:5945 trunc from 0x53898b6 to 0x2b98a80 2024-04-23T21:22:37.161 INFO:teuthology.orchestra.run.smithi111.stdout:5946 write 0x7a7ecb3 thru 0x7a82a9b (0x3de9 bytes) 2024-04-23T21:22:37.164 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:37.165 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:37.165 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:37.165 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:37.165+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:37.169 INFO:teuthology.orchestra.run.smithi111.stdout:5947 trunc from 0x7a82a9c to 0x33fd425 2024-04-23T21:22:37.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:37.186 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.202 INFO:teuthology.orchestra.run.smithi111.stdout:5950 writesame 0xc9b7031 thru 0xc9bfecd data_size 0x1(0x8e9d bytes) 2024-04-23T21:22:37.203 INFO:teuthology.orchestra.run.smithi111.stdout:5951 read 0xc8299a2 thru 0xc82d5c9 (0x3c28 bytes) 2024-04-23T21:22:37.203 INFO:teuthology.orchestra.run.smithi111.stdout:5952 writesame 0x51c2e6 thru 0x51ebc3 data_size 0x1(0x28de bytes) 2024-04-23T21:22:37.209 INFO:teuthology.orchestra.run.smithi111.stdout:5953 read 0x4b14f98 thru 0x4b2034d (0xb3b6 bytes) 2024-04-23T21:22:37.214 INFO:teuthology.orchestra.run.smithi111.stdout:5957 write 0xcdc1af0 thru 0xcdcca1f (0xaf30 bytes) 2024-04-23T21:22:37.226 INFO:teuthology.orchestra.run.smithi111.stdout:5958 trunc from 0xcdcca20 to 0x5b89ddb 2024-04-23T21:22:37.242 INFO:teuthology.orchestra.run.smithi111.stdout:5959 trunc from 0x5b89ddb to 0x102226c 2024-04-23T21:22:37.254 INFO:teuthology.orchestra.run.smithi111.stdout:5960 writesame 0x10d6811 thru 0x10de736 data_size 0x32(0x7f26 bytes) 2024-04-23T21:22:37.266 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:37.266+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:37.274 INFO:teuthology.orchestra.run.smithi111.stdout:5961 writesame 0xcf57d55 thru 0xcf5e996 data_size 0x1f(0x6c42 bytes) 2024-04-23T21:22:37.282 INFO:teuthology.orchestra.run.smithi111.stdout:5962 write 0xeadb271 thru 0xeae5585 (0xa315 bytes) 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 32.155335614000002, 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.355361869999999, 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:22:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.355339211, 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.355312691000002, 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.355292627000001, 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:22:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:22:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354999957, 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:22:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:22:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:22:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:22:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354990052000002, 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354687272, 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:22:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:22:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354650591999999, 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:22:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354628969, 2024-04-23T21:22:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:22:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:22:37.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354608723999998, 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.354407407, 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.331 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:22:37.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353972735999999, 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.335 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.336 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.337 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.338 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.339 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353705910999999, 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:37.340 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:22:37.341 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:22:37.342 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.343 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353683709999999, 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.344 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.345 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.346 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.347 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353664121000001, 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.348 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:22:37.350 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.351 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353325809000001, 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:22:37.352 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:22:37.353 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:22:37.354 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:22:37.355 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353305227, 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.356 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.357 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.358 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.359 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353286203, 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.360 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.361 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:22:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.363 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.353060038999999, 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:37.364 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:22:37.366 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:22:37.367 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:37.368 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5963 write 0xa3a2dfd thru 0xa3abf8b (0x918f bytes) 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5964 write 0x624bd58 thru 0x624fee6 (0x418f bytes) 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5965 write 0xba5636 thru 0xbb490e (0xf2d9 bytes) 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5967 writesame 0x84bc504 thru 0x84c4bed data_size 0x7(0x86ea bytes) 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5969 punch from 0x3574a86 to 0x3582be1, (0xe15b bytes) 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.370 INFO:teuthology.orchestra.run.smithi111.stdout:5971 clone 49 order 22 su 1048576 sc 6 2024-04-23T21:22:37.371 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:37.370+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:37.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:37.379 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T21:22:37.390 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:22:37.395 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-23T21:22:37.459 INFO:teuthology.orchestra.run.smithi111.stdout:truncating image image_client.0-clone48 from 0xeae5586 (overlap 0x43b02) to 0x0 2024-04-23T21:22:37.470 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:37.470+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:37.489 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:37.490 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:37.490 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:37.490 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:37.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:37.544 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #47, image image_client.0-clone47 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent48 2024-04-23T21:22:37.571 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:37.571+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:37.672 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:37.672+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:37.688 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:37.688 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:37.688 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:37.688 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:37.697 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:37.773 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:37.773+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:37.799 INFO:teuthology.orchestra.run.smithi111.stdout:5973 writesame 0xaddc41f thru 0xadeabd8 data_size 0x3(0xe7ba bytes) 2024-04-23T21:22:37.804 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 48.863880045999998, 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.805 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:37.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034481116999999, 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.810 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:37.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:22:37.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:22:37.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034458183000002, 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.814 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.815 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.816 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:22:37.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:37.818 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034432863999999, 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:37.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.822 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034414145, 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:22:37.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.825 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034371463999999, 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.827 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:37.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:22:37.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:37.832 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034345324, 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.833 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.834 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:22:37.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034327405999999, 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:22:37.837 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:22:37.839 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.840 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:22:37.841 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034296027, 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.842 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:22:37.844 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:22:37.845 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034278898, 2024-04-23T21:22:37.846 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:37.847 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:22:37.848 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:22:37.849 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034250360000001, 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.851 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:37.852 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:22:37.853 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.854 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034207375000001, 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.855 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:37.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034180138, 2024-04-23T21:22:37.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:22:37.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:22:37.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.863 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034156351, 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:37.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:22:37.865 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:22:37.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.867 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034131109, 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:22:37.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:22:37.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034112343, 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.872 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:22:37.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:22:37.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034055382999998, 2024-04-23T21:22:37.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034033649000001, 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:22:37.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 24.034009428000001, 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.885 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 4.6047403730000003, 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:22:37.889 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:37.895 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:37.897 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5975 punch from 0x6f20afc to 0x6f2ccde, (0xc1e2 bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5976 writesame 0x5351842 thru 0x5352b01 data_size 0x32(0x12c0 bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5977 write 0xe69326d thru 0xe696c1b (0x39af bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5978 write 0x9f292f2 thru 0x9f335ba (0xa2c9 bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5979 trunc from 0xeae5586 to 0x1c1f27b 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5981 trunc from 0x1c1f27b to 0xce7166a 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5982 read 0xb7a3235 thru 0xb7a43da (0x11a6 bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5984 read 0xcae011a thru 0xcaebca6 (0xbb8d bytes) 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.899 INFO:teuthology.orchestra.run.smithi111.stdout:5986 write 0xb0882 thru 0xb87d5 (0x7f54 bytes) 2024-04-23T21:22:37.900 INFO:teuthology.orchestra.run.smithi111.stdout:5987 read 0x38f49d3 thru 0x38fb75a (0x6d88 bytes) 2024-04-23T21:22:37.900 INFO:teuthology.orchestra.run.smithi111.stdout:5988 write 0x55ba704 thru 0x55c7554 (0xce51 bytes) 2024-04-23T21:22:37.900 INFO:teuthology.orchestra.run.smithi111.stdout:5989 write 0x1750b1b thru 0x175d401 (0xc8e7 bytes) 2024-04-23T21:22:37.900 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:37.897+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:37.900 INFO:teuthology.orchestra.run.smithi111.stdout:5990 write 0x878c6ba thru 0x878e29d (0x1be4 bytes) 2024-04-23T21:22:37.903 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.903 INFO:teuthology.orchestra.run.smithi111.stdout:5992 read 0xb4296ee thru 0xb433499 (0x9dac bytes) 2024-04-23T21:22:37.903 INFO:teuthology.orchestra.run.smithi111.stdout:5994 writesame 0x61932b8 thru 0x6194b08 data_size 0x19(0x1851 bytes) 2024-04-23T21:22:37.906 INFO:teuthology.orchestra.run.smithi111.stdout:5995 read 0x7e45817 thru 0x7e4f1e2 (0x99cc bytes) 2024-04-23T21:22:37.907 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.907 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.907 INFO:teuthology.orchestra.run.smithi111.stdout:5999 write 0x3b498a7 thru 0x3b4c2ca (0x2a24 bytes) 2024-04-23T21:22:37.910 INFO:teuthology.orchestra.run.smithi111.stdout:compare and write disabled 2024-04-23T21:22:37.924 INFO:teuthology.orchestra.run.smithi111.stdout:checking clone #48, image image_client.0-clone48 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent49 2024-04-23T21:22:37.997 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:37.997+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:38.010 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.010 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:38.010 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:38.011 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.020 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:38.098 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:38.099+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:38.163 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:38.163 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.172 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:38.216 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:38.216+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845484329000001, 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:38.276 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.280 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845461517999993, 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:38.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.282 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.283 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.284 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845441907999998, 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.285 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:38.286 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:38.287 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:38.288 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845423045000004, 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.289 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845403750000003, 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.293 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.294 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:38.295 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:38.296 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845377189000004, 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.297 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.298 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.299 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:38.300 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845335958999996, 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:38.301 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.302 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.304 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:38.305 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845302231999995, 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.306 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:38.307 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.308 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.309 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845278823000001, 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:38.310 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:38.311 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:38.312 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.313 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845245573, 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.314 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.315 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.316 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.317 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845222179999993, 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.318 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:38.319 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:38.320 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.321 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845201630999995, 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.322 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:38.323 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.324 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.325 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.845175870999995, 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.326 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:38.327 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.328 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:38.329 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.844337855000006, 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.330 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.331 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.332 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.333 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.844314523999998, 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.334 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.335 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.336 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.337 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.843427231000007, 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.338 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:38.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.843193651000007, 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.346 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.843076952999994, 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:38.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.842702539000001, 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.354 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.841345653999994, 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.355 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.358 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.359 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.359 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:38.361 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:38.360+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:38.378 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T21:22:38.379 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:22:38.400 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-23T21:22:38.461 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:38.460+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:38.499 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.499 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:38.499 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:38.499 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.508 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:38.562 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:38.562+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:38.632 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.632 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:38.632 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:38.632 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:38.632 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:38.663 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:38.664+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:38.746 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.515020118000002, 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.747 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:38.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:38.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514990451000003, 2024-04-23T21:22:38.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.755 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514971043000003, 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.756 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514949235000003, 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.760 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:38.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514929631999998, 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:38.764 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:38.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:38.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514910026999999, 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.769 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:38.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:38.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514889185999998, 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.773 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:38.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:38.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:38.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514859921000003, 2024-04-23T21:22:38.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:38.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:38.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.514828727999998, 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:38.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.706745716999997, 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.786 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.787 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:38.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703517499, 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703495117999999, 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.795 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:38.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:38.797 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:38.798 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703475840000003, 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.799 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.800 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:38.802 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703458406000003, 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:38.803 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.804 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.805 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.806 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703440956000001, 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:38.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:38.810 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703423581000003, 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.812 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.813 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.814 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.815 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703405539000002, 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.816 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.817 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.818 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.703380418000002, 2024-04-23T21:22:38.820 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.822 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.823 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.824 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 50.701260845999997, 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:38.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.826 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.827 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:38.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 38.361637854999998, 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:38.829 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:38.830 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:38.831 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:38.832 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:38.833 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:38.835 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:38.835+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:38.936 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:38.936+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:38.938 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:38.938 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:38.938 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:38.938 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:38.947 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:38.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:39.037 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:39.037+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:39.093 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:39.093 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:39.093 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:39.093 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:39.094 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:39.103 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:39.109 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:39.109 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:39.127 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:39.127 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:39.137 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:39.138+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:39.144 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:39.144 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:39.161 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:39.161 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:39.178 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:39.179 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:39.196 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:39.196 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:39.212 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.212 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.214 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:39.214 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:39.221 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:39.232 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:39.232 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:39.238 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:39.238+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:39.326 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.326 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:39.339 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:39.340+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:39.389 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T21:22:39.405 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-23T21:22:39.446 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:39.446+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:39.447 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.447 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:39.540 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:39.541+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:39.564 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.565 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:39.641 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:39.641+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:39.676 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.676 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.685 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:39.742 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:39.742+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:39.787 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.788 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.796 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:39.842 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:39.843+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:39.898 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:39.898 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:39.907 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:39.951 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:39.952+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:40.009 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.010 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:40.044 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:40.045+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:40.146 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:40.147+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:40.160 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.160 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.169 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:40.247 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:40.248+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:40.271 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.271 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.280 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:40.364 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:40.365+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:40.398 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T21:22:40.402 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:22:40.417 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-23T21:22:40.433 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.433 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:40.459 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:40.459+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:40.550 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:40.550+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:40.588 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.588 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:40.650 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:40.651+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:40.699 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.699 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.707 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:40.752 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:40.752+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:40.810 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.810 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.819 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:40.852 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:40.853+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:40.921 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:40.921 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:40.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:40.973 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:40.974+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:41.031 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:41.031 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:41.040 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:41.053 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:41.054+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:41.154 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:41.155+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:41.182 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:41.182 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:41.191 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:41.243 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:41.288 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:41.289+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:41.295 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:41.295 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:41.305 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:41.365 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:41.365 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:41.365 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:41.366+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:41.382 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:41.383 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:41.399 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:41.399 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:41.401 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:22:41.402 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T21:22:41.420 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-23T21:22:41.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:41.429 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:41.429 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:41.448 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:41.448 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:41.457 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:41.459+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:41.459 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:41.459 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:41.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.029077888000003, 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:41.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:41.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.029055076999995, 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.464 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.468 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.029035467, 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:41.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.029016604000006, 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.473 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:41.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:41.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028997309000005, 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:41.477 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:41.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:41.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028970748000006, 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:41.485 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028929517999998, 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.486 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.490 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028895790999997, 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:41.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:41.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028872382000003, 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:41.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:41.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028839132000002, 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:41.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:41.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028815738999995, 2024-04-23T21:22:41.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:41.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.507 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028795189999997, 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.511 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.028769429999997, 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.512 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:41.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.027931413999994, 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.516 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:41.520 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.027908083, 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:41.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.027020789999995, 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:41.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.026787209999995, 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.529 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:41.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.026670511999995, 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.537 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.026296098000003, 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 72.024939212999996, 2024-04-23T21:22:41.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.546 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:41.548 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:41.548 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:41.549 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:41.549 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:41.549 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:41.550 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:41.550 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:41.589 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:41.590+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:41.659 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:41.660+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:41.691 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:41.691 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:41.691 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:41.691 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:41.701 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:41.732 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:41.733 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:41.745 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-23T21:22:41.608947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2385990941,"num_objects":2177,"num_object_clones":497,"num_object_copies":4354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":1004,"num_read":177156,"num_read_kb":1010829,"num_write":18022,"num_write_kb":600061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14802,"ondisk_log_size":14802,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":396268,"kb_used_data":97536,"kb_used_omap":15,"kb_used_meta":298608,"kb_avail":660206612,"statfs":{"total":676457349120,"available":676051570688,"internally_reserved":0,"allocated":99876864,"data_stored":926663705,"data_compressed":55403008,"data_compressed_allocated":60747776,"data_compressed_original":895380764,"omap_allocated":16311,"internal_metadata":305774665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70101247,"num_objects":56,"num_object_clones":19,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":30,"num_read":5496,"num_read_kb":14941,"num_write":681,"num_write_kb":25681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000699"},"pg_stats":[{"pgid":"3.1f","version":"92'418","reported_seq":4067,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387746+0000","last_change":"2024-04-23T21:22:38.387900+0000","last_active":"2024-04-23T21:22:38.387746+0000","last_peered":"2024-04-23T21:22:38.387746+0000","last_clean":"2024-04-23T21:22:38.387746+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:38.387746+0000","last_undegraded":"2024-04-23T21:22:38.387746+0000","last_fullsized":"2024-04-23T21:22:38.387746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:46:08.750064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000807083,"stat_sum":{"num_bytes":73070064,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5911,"num_read_kb":16706,"num_write":482,"num_write_kb":33855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1e","version":"94'374","reported_seq":6612,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440046+0000","last_change":"2024-04-23T21:22:38.440210+0000","last_active":"2024-04-23T21:22:38.440046+0000","last_peered":"2024-04-23T21:22:38.440046+0000","last_clean":"2024-04-23T21:22:38.440046+0000","last_became_active":"2024-04-23T21:22:33.186559+0000","last_became_peered":"2024-04-23T21:22:33.186559+0000","last_unstale":"2024-04-23T21:22:38.440046+0000","last_undegraded":"2024-04-23T21:22:38.440046+0000","last_fullsized":"2024-04-23T21:22:38.440046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:32.760937+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0018364259999999999,"stat_sum":{"num_bytes":76868783,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":32,"num_read":6375,"num_read_kb":47205,"num_write":430,"num_write_kb":24897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1d","version":"83'279","reported_seq":8176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387401+0000","last_change":"2024-04-23T21:22:38.387584+0000","last_active":"2024-04-23T21:22:38.387401+0000","last_peered":"2024-04-23T21:22:38.387401+0000","last_clean":"2024-04-23T21:22:38.387401+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:38.387401+0000","last_undegraded":"2024-04-23T21:22:38.387401+0000","last_fullsized":"2024-04-23T21:22:38.387401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:32.941280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070416999999999995,"stat_sum":{"num_bytes":46779874,"num_objects":65,"num_object_clones":14,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":32,"num_read":8295,"num_read_kb":31836,"num_write":262,"num_write_kb":7752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1c","version":"94'397","reported_seq":2921,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.473422+0000","last_change":"2024-04-23T21:22:38.387273+0000","last_active":"2024-04-23T21:22:38.473422+0000","last_peered":"2024-04-23T21:22:38.473422+0000","last_clean":"2024-04-23T21:22:38.473422+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:38.473422+0000","last_undegraded":"2024-04-23T21:22:38.473422+0000","last_fullsized":"2024-04-23T21:22:38.473422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:35.225413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000435031,"stat_sum":{"num_bytes":69525595,"num_objects":79,"num_object_clones":21,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":39,"num_read":2897,"num_read_kb":24662,"num_write":465,"num_write_kb":16868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1b","version":"91'327","reported_seq":9994,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389864+0000","last_change":"2024-04-23T21:22:38.390268+0000","last_active":"2024-04-23T21:22:38.389864+0000","last_peered":"2024-04-23T21:22:38.389864+0000","last_clean":"2024-04-23T21:22:38.389864+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:38.389864+0000","last_undegraded":"2024-04-23T21:22:38.389864+0000","last_fullsized":"2024-04-23T21:22:38.389864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:11.320083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080390080000000003,"stat_sum":{"num_bytes":114321291,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":31,"num_read":10911,"num_read_kb":51379,"num_write":339,"num_write_kb":56482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1a","version":"94'781","reported_seq":5484,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.482706+0000","last_change":"2024-04-23T21:22:38.389806+0000","last_active":"2024-04-23T21:22:38.482706+0000","last_peered":"2024-04-23T21:22:38.482706+0000","last_clean":"2024-04-23T21:22:38.482706+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:38.482706+0000","last_undegraded":"2024-04-23T21:22:38.482706+0000","last_fullsized":"2024-04-23T21:22:38.482706+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:44.982850+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0079933109999999995,"stat_sum":{"num_bytes":59811408,"num_objects":61,"num_object_clones":11,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":29,"num_read":8983,"num_read_kb":51586,"num_write":989,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.19","version":"91'750","reported_seq":2570,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380186+0000","last_change":"2024-04-23T21:22:34.187783+0000","last_active":"2024-04-23T21:22:37.380186+0000","last_peered":"2024-04-23T21:22:37.380186+0000","last_clean":"2024-04-23T21:22:37.380186+0000","last_became_active":"2024-04-23T21:22:34.185924+0000","last_became_peered":"2024-04-23T21:22:34.185924+0000","last_unstale":"2024-04-23T21:22:37.380186+0000","last_undegraded":"2024-04-23T21:22:37.380186+0000","last_fullsized":"2024-04-23T21:22:37.380186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:22:23.821959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":44822653,"num_objects":67,"num_object_clones":14,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":35,"num_read":4926,"num_read_kb":14456,"num_write":1053,"num_write_kb":9858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"92'635","reported_seq":5403,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380301+0000","last_change":"2024-04-23T21:22:33.190596+0000","last_active":"2024-04-23T21:22:37.380301+0000","last_peered":"2024-04-23T21:22:37.380301+0000","last_clean":"2024-04-23T21:22:37.380301+0000","last_became_active":"2024-04-23T21:22:33.189427+0000","last_became_peered":"2024-04-23T21:22:33.189427+0000","last_unstale":"2024-04-23T21:22:37.380301+0000","last_undegraded":"2024-04-23T21:22:37.380301+0000","last_fullsized":"2024-04-23T21:22:37.380301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:31.040341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":44931716,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":37,"num_read":7690,"num_read_kb":24842,"num_write":813,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":5309940,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"93'499","reported_seq":2648,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387837+0000","last_change":"2024-04-23T21:22:38.387987+0000","last_active":"2024-04-23T21:22:38.387837+0000","last_peered":"2024-04-23T21:22:38.387837+0000","last_clean":"2024-04-23T21:22:38.387837+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:38.387837+0000","last_undegraded":"2024-04-23T21:22:38.387837+0000","last_fullsized":"2024-04-23T21:22:38.387837+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:54:49.539746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086574500000000001,"stat_sum":{"num_bytes":59572632,"num_objects":74,"num_object_clones":13,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":3820,"num_read_kb":30740,"num_write":539,"num_write_kb":28205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.16","version":"92'394","reported_seq":5056,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.476685+0000","last_change":"2024-04-23T21:22:38.387802+0000","last_active":"2024-04-23T21:22:38.476685+0000","last_peered":"2024-04-23T21:22:38.476685+0000","last_clean":"2024-04-23T21:22:38.476685+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:38.476685+0000","last_undegraded":"2024-04-23T21:22:38.476685+0000","last_fullsized":"2024-04-23T21:22:38.476685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:25.134906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087084800000000002,"stat_sum":{"num_bytes":61828228,"num_objects":55,"num_object_clones":11,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5990,"num_read_kb":36089,"num_write":431,"num_write_kb":7286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.15","version":"92'333","reported_seq":4406,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384339+0000","last_change":"2024-04-23T21:22:38.384573+0000","last_active":"2024-04-23T21:22:38.384339+0000","last_peered":"2024-04-23T21:22:38.384339+0000","last_clean":"2024-04-23T21:22:38.384339+0000","last_became_active":"2024-04-23T21:22:33.191679+0000","last_became_peered":"2024-04-23T21:22:33.191679+0000","last_unstale":"2024-04-23T21:22:38.384339+0000","last_undegraded":"2024-04-23T21:22:38.384339+0000","last_fullsized":"2024-04-23T21:22:38.384339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:05.849253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001744839,"stat_sum":{"num_bytes":74091838,"num_objects":72,"num_object_clones":18,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":33,"num_read":4322,"num_read_kb":26762,"num_write":336,"num_write_kb":9614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":5579029,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.14","version":"92'592","reported_seq":5561,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.390319+0000","last_change":"2024-04-23T21:22:38.394763+0000","last_active":"2024-04-23T21:22:38.390319+0000","last_peered":"2024-04-23T21:22:38.390319+0000","last_clean":"2024-04-23T21:22:38.390319+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:38.390319+0000","last_undegraded":"2024-04-23T21:22:38.390319+0000","last_fullsized":"2024-04-23T21:22:38.390319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:05:13.327032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012237974,"stat_sum":{"num_bytes":110248973,"num_objects":66,"num_object_clones":13,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":7406,"num_read_kb":52041,"num_write":742,"num_write_kb":17070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.438285+0000","last_change":"2024-04-23T21:22:30.359764+0000","last_active":"2024-04-23T21:22:38.438285+0000","last_peered":"2024-04-23T21:22:38.438285+0000","last_clean":"2024-04-23T21:22:38.438285+0000","last_became_active":"2024-04-23T21:22:30.358837+0000","last_became_peered":"2024-04-23T21:22:30.358837+0000","last_unstale":"2024-04-23T21:22:38.438285+0000","last_undegraded":"2024-04-23T21:22:38.438285+0000","last_fullsized":"2024-04-23T21:22:38.438285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:38.092515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"93'401","reported_seq":2610,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.466632+0000","last_change":"2024-04-23T21:22:38.387701+0000","last_active":"2024-04-23T21:22:38.466632+0000","last_peered":"2024-04-23T21:22:38.466632+0000","last_clean":"2024-04-23T21:22:38.466632+0000","last_became_active":"2024-04-23T21:22:33.186422+0000","last_became_peered":"2024-04-23T21:22:33.186422+0000","last_unstale":"2024-04-23T21:22:38.466632+0000","last_undegraded":"2024-04-23T21:22:38.466632+0000","last_fullsized":"2024-04-23T21:22:38.466632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:36.279822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578999999999997,"stat_sum":{"num_bytes":62894375,"num_objects":70,"num_object_clones":17,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":26,"num_read":2607,"num_read_kb":19018,"num_write":480,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":10723491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382567+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:38.382567+0000","last_peered":"2024-04-23T21:22:38.382567+0000","last_clean":"2024-04-23T21:22:38.382567+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:38.382567+0000","last_undegraded":"2024-04-23T21:22:38.382567+0000","last_fullsized":"2024-04-23T21:22:38.382567+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"93'351","reported_seq":3853,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.939487+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:37.939487+0000","last_peered":"2024-04-23T21:22:37.939487+0000","last_clean":"2024-04-23T21:22:37.939487+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:37.939487+0000","last_undegraded":"2024-04-23T21:22:37.939487+0000","last_fullsized":"2024-04-23T21:22:37.939487+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:07.398443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":86833958,"num_objects":62,"num_object_clones":17,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":26,"num_read":4697,"num_read_kb":34568,"num_write":316,"num_write_kb":17871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382318+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:38.382318+0000","last_peered":"2024-04-23T21:22:38.382318+0000","last_clean":"2024-04-23T21:22:38.382318+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:38.382318+0000","last_undegraded":"2024-04-23T21:22:38.382318+0000","last_fullsized":"2024-04-23T21:22:38.382318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"93'360","reported_seq":3403,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384073+0000","last_change":"2024-04-23T21:22:38.384310+0000","last_active":"2024-04-23T21:22:38.384073+0000","last_peered":"2024-04-23T21:22:38.384073+0000","last_clean":"2024-04-23T21:22:38.384073+0000","last_became_active":"2024-04-23T21:22:33.191620+0000","last_became_peered":"2024-04-23T21:22:33.191620+0000","last_unstale":"2024-04-23T21:22:38.384073+0000","last_undegraded":"2024-04-23T21:22:38.384073+0000","last_fullsized":"2024-04-23T21:22:38.384073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:04:04.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00161089,"stat_sum":{"num_bytes":61310486,"num_objects":66,"num_object_clones":11,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":3589,"num_read_kb":24366,"num_write":435,"num_write_kb":11827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":9761436,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386938+0000","last_change":"2024-04-23T21:22:30.359345+0000","last_active":"2024-04-23T21:22:38.386938+0000","last_peered":"2024-04-23T21:22:38.386938+0000","last_clean":"2024-04-23T21:22:38.386938+0000","last_became_active":"2024-04-23T21:22:30.359070+0000","last_became_peered":"2024-04-23T21:22:30.359070+0000","last_unstale":"2024-04-23T21:22:38.386938+0000","last_undegraded":"2024-04-23T21:22:38.386938+0000","last_fullsized":"2024-04-23T21:22:38.386938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:52.085200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"92'308","reported_seq":4025,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380217+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:37.380217+0000","last_peered":"2024-04-23T21:22:37.380217+0000","last_clean":"2024-04-23T21:22:37.380217+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:37.380217+0000","last_undegraded":"2024-04-23T21:22:37.380217+0000","last_fullsized":"2024-04-23T21:22:37.380217+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:27.966227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":90459570,"num_objects":79,"num_object_clones":19,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":40,"num_read":5935,"num_read_kb":33564,"num_write":327,"num_write_kb":18238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"79'345","reported_seq":8300,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382015+0000","last_change":"2024-04-23T21:22:38.382561+0000","last_active":"2024-04-23T21:22:38.382015+0000","last_peered":"2024-04-23T21:22:38.382015+0000","last_clean":"2024-04-23T21:22:38.382015+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:38.382015+0000","last_undegraded":"2024-04-23T21:22:38.382015+0000","last_fullsized":"2024-04-23T21:22:38.382015+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:31:27.059067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074128699999999998,"stat_sum":{"num_bytes":149341517,"num_objects":79,"num_object_clones":20,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":32,"num_read":8275,"num_read_kb":76033,"num_write":455,"num_write_kb":35588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":323,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382260+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:38.382260+0000","last_peered":"2024-04-23T21:22:38.382260+0000","last_clean":"2024-04-23T21:22:38.382260+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:38.382260+0000","last_undegraded":"2024-04-23T21:22:38.382260+0000","last_fullsized":"2024-04-23T21:22:38.382260+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380159+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:37.380159+0000","last_peered":"2024-04-23T21:22:37.380159+0000","last_clean":"2024-04-23T21:22:37.380159+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:37.380159+0000","last_undegraded":"2024-04-23T21:22:37.380159+0000","last_fullsized":"2024-04-23T21:22:37.380159+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'287","reported_seq":915,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.896604+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:37.896604+0000","last_peered":"2024-04-23T21:22:37.896604+0000","last_clean":"2024-04-23T21:22:37.896604+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:37.896604+0000","last_undegraded":"2024-04-23T21:22:37.896604+0000","last_fullsized":"2024-04-23T21:22:37.896604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":32288121,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":30,"num_read":3345,"num_read_kb":9678,"num_write":326,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382515+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:38.382515+0000","last_peered":"2024-04-23T21:22:38.382515+0000","last_clean":"2024-04-23T21:22:38.382515+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:38.382515+0000","last_undegraded":"2024-04-23T21:22:38.382515+0000","last_fullsized":"2024-04-23T21:22:38.382515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'651","reported_seq":7711,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483868+0000","last_change":"2024-04-23T21:22:38.395147+0000","last_active":"2024-04-23T21:22:38.483868+0000","last_peered":"2024-04-23T21:22:38.483868+0000","last_clean":"2024-04-23T21:22:38.483868+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:38.483868+0000","last_undegraded":"2024-04-23T21:22:38.483868+0000","last_fullsized":"2024-04-23T21:22:38.483868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:24.395971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359706,"stat_sum":{"num_bytes":88289494,"num_objects":85,"num_object_clones":22,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":34,"num_read":9955,"num_read_kb":57479,"num_write":910,"num_write_kb":33390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386737+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:38.386737+0000","last_peered":"2024-04-23T21:22:38.386737+0000","last_clean":"2024-04-23T21:22:38.386737+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:38.386737+0000","last_undegraded":"2024-04-23T21:22:38.386737+0000","last_fullsized":"2024-04-23T21:22:38.386737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'515","reported_seq":4728,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387255+0000","last_change":"2024-04-23T21:22:38.387429+0000","last_active":"2024-04-23T21:22:38.387255+0000","last_peered":"2024-04-23T21:22:38.387255+0000","last_clean":"2024-04-23T21:22:38.387255+0000","last_became_active":"2024-04-23T21:22:33.185729+0000","last_became_peered":"2024-04-23T21:22:33.185729+0000","last_unstale":"2024-04-23T21:22:38.387255+0000","last_undegraded":"2024-04-23T21:22:38.387255+0000","last_fullsized":"2024-04-23T21:22:38.387255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:17.916282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058378599999999998,"stat_sum":{"num_bytes":59185175,"num_objects":52,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":24,"num_read":6960,"num_read_kb":14315,"num_write":677,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382362+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:38.382362+0000","last_peered":"2024-04-23T21:22:38.382362+0000","last_clean":"2024-04-23T21:22:38.382362+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:38.382362+0000","last_undegraded":"2024-04-23T21:22:38.382362+0000","last_fullsized":"2024-04-23T21:22:38.382362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"93'241","reported_seq":2490,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.602023+0000","last_change":"2024-04-23T21:22:33.191726+0000","last_active":"2024-04-23T21:22:37.602023+0000","last_peered":"2024-04-23T21:22:37.602023+0000","last_clean":"2024-04-23T21:22:37.602023+0000","last_became_active":"2024-04-23T21:22:33.190419+0000","last_became_peered":"2024-04-23T21:22:33.190419+0000","last_unstale":"2024-04-23T21:22:37.602023+0000","last_undegraded":"2024-04-23T21:22:37.602023+0000","last_fullsized":"2024-04-23T21:22:37.602023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:45.249930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":35042921,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":35,"num_read":2161,"num_read_kb":15419,"num_write":257,"num_write_kb":12006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":13465209,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'296","reported_seq":7195,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.494395+0000","last_change":"2024-04-23T21:22:38.389596+0000","last_active":"2024-04-23T21:22:38.494395+0000","last_peered":"2024-04-23T21:22:38.494395+0000","last_clean":"2024-04-23T21:22:38.494395+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:38.494395+0000","last_undegraded":"2024-04-23T21:22:38.494395+0000","last_fullsized":"2024-04-23T21:22:38.494395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:37.705357+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077740689999999998,"stat_sum":{"num_bytes":104400617,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":34,"num_read":7010,"num_read_kb":46714,"num_write":329,"num_write_kb":15720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.a","version":"94'460","reported_seq":3385,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440408+0000","last_change":"2024-04-23T21:22:38.440408+0000","last_active":"2024-04-23T21:22:38.440408+0000","last_peered":"2024-04-23T21:22:38.440408+0000","last_clean":"2024-04-23T21:22:38.440408+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:38.440408+0000","last_undegraded":"2024-04-23T21:22:38.440408+0000","last_fullsized":"2024-04-23T21:22:38.440408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:03.627859+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001926042,"stat_sum":{"num_bytes":41021335,"num_objects":59,"num_object_clones":10,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":28,"num_read":5295,"num_read_kb":12681,"num_write":457,"num_write_kb":14975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.b","version":"94'402","reported_seq":2272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496515+0000","last_change":"2024-04-23T21:22:38.396815+0000","last_active":"2024-04-23T21:22:38.496515+0000","last_peered":"2024-04-23T21:22:38.496515+0000","last_clean":"2024-04-23T21:22:38.496515+0000","last_became_active":"2024-04-23T21:22:33.192667+0000","last_became_peered":"2024-04-23T21:22:33.192667+0000","last_unstale":"2024-04-23T21:22:38.496515+0000","last_undegraded":"2024-04-23T21:22:38.496515+0000","last_fullsized":"2024-04-23T21:22:38.496515+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:24.257165+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.014784952000000001,"stat_sum":{"num_bytes":64284813,"num_objects":66,"num_object_clones":14,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":30,"num_read":2590,"num_read_kb":25806,"num_write":435,"num_write_kb":14302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":9997927,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.c","version":"92'351","reported_seq":1633,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387010+0000","last_change":"2024-04-23T21:22:38.387207+0000","last_active":"2024-04-23T21:22:38.387010+0000","last_peered":"2024-04-23T21:22:38.387010+0000","last_clean":"2024-04-23T21:22:38.387010+0000","last_became_active":"2024-04-23T21:22:33.185774+0000","last_became_peered":"2024-04-23T21:22:33.185774+0000","last_unstale":"2024-04-23T21:22:38.387010+0000","last_undegraded":"2024-04-23T21:22:38.387010+0000","last_fullsized":"2024-04-23T21:22:38.387010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:50:58.291001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000369157,"stat_sum":{"num_bytes":95852508,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":34,"num_read":1441,"num_read_kb":18659,"num_write":330,"num_write_kb":17730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.d","version":"94'264","reported_seq":2543,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389126+0000","last_change":"2024-04-23T21:22:38.389428+0000","last_active":"2024-04-23T21:22:38.389126+0000","last_peered":"2024-04-23T21:22:38.389126+0000","last_clean":"2024-04-23T21:22:38.389126+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:38.389126+0000","last_undegraded":"2024-04-23T21:22:38.389126+0000","last_fullsized":"2024-04-23T21:22:38.389126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:54:15.435558+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0063591359999999996,"stat_sum":{"num_bytes":45966183,"num_objects":54,"num_object_clones":9,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":29,"num_read":2501,"num_read_kb":11452,"num_write":248,"num_write_kb":11068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.e","version":"94'506","reported_seq":1920,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496079+0000","last_change":"2024-04-23T21:22:38.384963+0000","last_active":"2024-04-23T21:22:38.496079+0000","last_peered":"2024-04-23T21:22:38.496079+0000","last_clean":"2024-04-23T21:22:38.496079+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:38.496079+0000","last_undegraded":"2024-04-23T21:22:38.496079+0000","last_fullsized":"2024-04-23T21:22:38.496079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:09:46.962009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320499999999999,"stat_sum":{"num_bytes":48671382,"num_objects":55,"num_object_clones":10,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":27,"num_read":3448,"num_read_kb":11204,"num_write":630,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.f","version":"94'896","reported_seq":9954,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.395078+0000","last_change":"2024-04-23T21:22:38.395242+0000","last_active":"2024-04-23T21:22:38.395078+0000","last_peered":"2024-04-23T21:22:38.395078+0000","last_clean":"2024-04-23T21:22:38.395078+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:38.395078+0000","last_undegraded":"2024-04-23T21:22:38.395078+0000","last_fullsized":"2024-04-23T21:22:38.395078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:41:01.685567+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012161335000000001,"stat_sum":{"num_bytes":127130709,"num_objects":81,"num_object_clones":18,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":32,"num_read":13152,"num_read_kb":58727,"num_write":1272,"num_write_kb":16409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.10","version":"93'414","reported_seq":3099,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483088+0000","last_change":"2024-04-23T21:22:38.438886+0000","last_active":"2024-04-23T21:22:38.483088+0000","last_peered":"2024-04-23T21:22:38.483088+0000","last_clean":"2024-04-23T21:22:38.483088+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:38.483088+0000","last_undegraded":"2024-04-23T21:22:38.483088+0000","last_fullsized":"2024-04-23T21:22:38.483088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:47:55.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047995299999999998,"stat_sum":{"num_bytes":101761897,"num_objects":74,"num_object_clones":17,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":30,"num_read":4138,"num_read_kb":33523,"num_write":481,"num_write_kb":20694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.11","version":"94'253","reported_seq":2188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389732+0000","last_change":"2024-04-23T21:22:38.390017+0000","last_active":"2024-04-23T21:22:38.389732+0000","last_peered":"2024-04-23T21:22:38.389732+0000","last_clean":"2024-04-23T21:22:38.389732+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:38.389732+0000","last_undegraded":"2024-04-23T21:22:38.389732+0000","last_fullsized":"2024-04-23T21:22:38.389732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:53:16.375004+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077344930000000003,"stat_sum":{"num_bytes":91411180,"num_objects":66,"num_object_clones":17,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":27,"num_read":1874,"num_read_kb":20591,"num_write":283,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.12","version":"94'789","reported_seq":2928,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.454378+0000","last_change":"2024-04-23T21:22:38.384028+0000","last_active":"2024-04-23T21:22:38.454378+0000","last_peered":"2024-04-23T21:22:38.454378+0000","last_clean":"2024-04-23T21:22:38.454378+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:38.454378+0000","last_undegraded":"2024-04-23T21:22:38.454378+0000","last_fullsized":"2024-04-23T21:22:38.454378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:26:45.125365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014285649999999999,"stat_sum":{"num_bytes":80276131,"num_objects":72,"num_object_clones":12,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":27,"num_read":5353,"num_read_kb":39496,"num_write":1086,"num_write_kb":21634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.13","version":"93'816","reported_seq":2742,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382845+0000","last_change":"2024-04-23T21:22:38.383310+0000","last_active":"2024-04-23T21:22:38.382845+0000","last_peered":"2024-04-23T21:22:38.382845+0000","last_clean":"2024-04-23T21:22:38.382845+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:38.382845+0000","last_undegraded":"2024-04-23T21:22:38.382845+0000","last_fullsized":"2024-04-23T21:22:38.382845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:11:00.823606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084378100000000004,"stat_sum":{"num_bytes":83105146,"num_objects":70,"num_object_clones":14,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":5188,"num_read_kb":39132,"num_write":1248,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2385400573,"num_objects":2175,"num_object_clones":497,"num_object_copies":4350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":1004,"num_read":177040,"num_read_kb":1010729,"num_write":17863,"num_write_kb":597645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92680192,"data_stored":927288416,"data_compressed":55633033,"data_compressed_allocated":61050880,"data_compressed_original":899446117,"omap_allocated":0,"internal_metadata":0},"log_size":14685,"ondisk_log_size":14685,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65288,"kb_used_data":16060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94306552,"statfs":{"total":96636764160,"available":96569909248,"internally_reserved":0,"allocated":16445440,"data_stored":144462534,"data_compressed":9363404,"data_compressed_allocated":10027008,"data_compressed_original":139355245,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42648,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329192,"statfs":{"total":96636764160,"available":96593092608,"internally_reserved":0,"allocated":7475200,"data_stored":64982199,"data_compressed":3626218,"data_compressed_allocated":4145152,"data_compressed_original":62472853,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49644,"kb_used_data":10136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322196,"statfs":{"total":96636764160,"available":96585928704,"internally_reserved":0,"allocated":10379264,"data_stored":87504141,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83980,"kb_used_data":26436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94287860,"statfs":{"total":96636764160,"available":96550768640,"internally_reserved":0,"allocated":27070464,"data_stored":267140093,"data_compressed":15733233,"data_compressed_allocated":17158144,"data_compressed_original":258789727,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39128,"kb_used_data":7936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332712,"statfs":{"total":96636764160,"available":96596697088,"internally_reserved":0,"allocated":8126464,"data_stored":65599087,"data_compressed":3837804,"data_compressed_allocated":4386816,"data_compressed_original":62678344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528331,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46560,"kb_used_data":9876,"kb_used_omap":6,"kb_used_meta":36665,"kb_avail":94325280,"statfs":{"total":96636764160,"available":96589086720,"internally_reserved":0,"allocated":10113024,"data_stored":97825612,"data_compressed":6089726,"data_compressed_allocated":6176768,"data_compressed_original":94863024,"omap_allocated":6748,"internal_metadata":37545380},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69020,"kb_used_data":19792,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302820,"statfs":{"total":96636764160,"available":96566087680,"internally_reserved":0,"allocated":20267008,"data_stored":199150039,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19156992,"data_stored":198632385,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9138176,"data_stored":96862759,"data_compressed":6055668,"data_compressed_allocated":6131712,"data_compressed_original":94273200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":64762460,"data_compressed":3818707,"data_compressed_allocated":4358144,"data_compressed_original":62364230,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":6676463,"data_compressed":354993,"data_compressed_allocated":483328,"data_compressed_original":6245965,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":266049296,"data_compressed":15712561,"data_compressed_allocated":17125376,"data_compressed_original":258392415,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9117696,"data_stored":86851690,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6336512,"data_stored":64152842,"data_compressed":3607499,"data_compressed_allocated":4116480,"data_compressed_original":62166931,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15212544,"data_stored":143300521,"data_compressed":9330982,"data_compressed_allocated":9981952,"data_compressed_original":138781805,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:41.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:41.786 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:41.787+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:41.805 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:41.805 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:41.805 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:41.805 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:41.806 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:41.815 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:41.899 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:41.900+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:41.962 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:41.963+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:41.963 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:41.963 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:41.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:41.963 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732465337000001, 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.964 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:41.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732435670000001, 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.968 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:41.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:41.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732416262000001, 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:41.972 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.975 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:41.976 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732394454000001, 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:41.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.979 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.980 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732374851000003, 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.981 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:41.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:41.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:41.984 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732355245999997, 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.985 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:41.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:41.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732334405000003, 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.989 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.992 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:41.993 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732305140000001, 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:41.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:41.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:41.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.732273947000003, 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:41.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:42.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:42.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.924190936000002, 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.002 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920962717999998, 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.006 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:42.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:42.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920940336999998, 2024-04-23T21:22:42.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:42.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:42.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:42.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920921059000001, 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:42.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:42.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920903625000001, 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.020 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920886175, 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:42.024 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:42.028 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920868800000001, 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:42.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:42.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.032 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920850758, 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.920825637, 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 53.918706065000002, 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.044 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 41.579083074000003, 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:42.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:42.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:42.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:42.048 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.049 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.049 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.049 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:42.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:42.064 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:42.064+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:42.078 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:42.079 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:42.091 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-04-23T21:22:41.608947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2385990941,"num_objects":2177,"num_object_clones":497,"num_object_copies":4354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":1004,"num_read":177156,"num_read_kb":1010829,"num_write":18022,"num_write_kb":600061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14802,"ondisk_log_size":14802,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":396268,"kb_used_data":97536,"kb_used_omap":15,"kb_used_meta":298608,"kb_avail":660206612,"statfs":{"total":676457349120,"available":676051570688,"internally_reserved":0,"allocated":99876864,"data_stored":926663705,"data_compressed":55403008,"data_compressed_allocated":60747776,"data_compressed_original":895380764,"omap_allocated":16311,"internal_metadata":305774665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":70101247,"num_objects":56,"num_object_clones":19,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":30,"num_read":5496,"num_read_kb":14941,"num_write":681,"num_write_kb":25681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000699"},"pg_stats":[{"pgid":"3.1f","version":"92'418","reported_seq":4067,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387746+0000","last_change":"2024-04-23T21:22:38.387900+0000","last_active":"2024-04-23T21:22:38.387746+0000","last_peered":"2024-04-23T21:22:38.387746+0000","last_clean":"2024-04-23T21:22:38.387746+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:38.387746+0000","last_undegraded":"2024-04-23T21:22:38.387746+0000","last_fullsized":"2024-04-23T21:22:38.387746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:46:08.750064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000807083,"stat_sum":{"num_bytes":73070064,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5911,"num_read_kb":16706,"num_write":482,"num_write_kb":33855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1e","version":"94'374","reported_seq":6612,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440046+0000","last_change":"2024-04-23T21:22:38.440210+0000","last_active":"2024-04-23T21:22:38.440046+0000","last_peered":"2024-04-23T21:22:38.440046+0000","last_clean":"2024-04-23T21:22:38.440046+0000","last_became_active":"2024-04-23T21:22:33.186559+0000","last_became_peered":"2024-04-23T21:22:33.186559+0000","last_unstale":"2024-04-23T21:22:38.440046+0000","last_undegraded":"2024-04-23T21:22:38.440046+0000","last_fullsized":"2024-04-23T21:22:38.440046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:32.760937+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0018364259999999999,"stat_sum":{"num_bytes":76868783,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":32,"num_read":6375,"num_read_kb":47205,"num_write":430,"num_write_kb":24897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1d","version":"83'279","reported_seq":8176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387401+0000","last_change":"2024-04-23T21:22:38.387584+0000","last_active":"2024-04-23T21:22:38.387401+0000","last_peered":"2024-04-23T21:22:38.387401+0000","last_clean":"2024-04-23T21:22:38.387401+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:38.387401+0000","last_undegraded":"2024-04-23T21:22:38.387401+0000","last_fullsized":"2024-04-23T21:22:38.387401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:32.941280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070416999999999995,"stat_sum":{"num_bytes":46779874,"num_objects":65,"num_object_clones":14,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":32,"num_read":8295,"num_read_kb":31836,"num_write":262,"num_write_kb":7752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1c","version":"94'397","reported_seq":2921,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.473422+0000","last_change":"2024-04-23T21:22:38.387273+0000","last_active":"2024-04-23T21:22:38.473422+0000","last_peered":"2024-04-23T21:22:38.473422+0000","last_clean":"2024-04-23T21:22:38.473422+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:38.473422+0000","last_undegraded":"2024-04-23T21:22:38.473422+0000","last_fullsized":"2024-04-23T21:22:38.473422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:35.225413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000435031,"stat_sum":{"num_bytes":69525595,"num_objects":79,"num_object_clones":21,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":39,"num_read":2897,"num_read_kb":24662,"num_write":465,"num_write_kb":16868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1b","version":"91'327","reported_seq":9994,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389864+0000","last_change":"2024-04-23T21:22:38.390268+0000","last_active":"2024-04-23T21:22:38.389864+0000","last_peered":"2024-04-23T21:22:38.389864+0000","last_clean":"2024-04-23T21:22:38.389864+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:38.389864+0000","last_undegraded":"2024-04-23T21:22:38.389864+0000","last_fullsized":"2024-04-23T21:22:38.389864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:11.320083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080390080000000003,"stat_sum":{"num_bytes":114321291,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":31,"num_read":10911,"num_read_kb":51379,"num_write":339,"num_write_kb":56482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1a","version":"94'781","reported_seq":5484,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.482706+0000","last_change":"2024-04-23T21:22:38.389806+0000","last_active":"2024-04-23T21:22:38.482706+0000","last_peered":"2024-04-23T21:22:38.482706+0000","last_clean":"2024-04-23T21:22:38.482706+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:38.482706+0000","last_undegraded":"2024-04-23T21:22:38.482706+0000","last_fullsized":"2024-04-23T21:22:38.482706+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:44.982850+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0079933109999999995,"stat_sum":{"num_bytes":59811408,"num_objects":61,"num_object_clones":11,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":29,"num_read":8983,"num_read_kb":51586,"num_write":989,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.19","version":"91'750","reported_seq":2570,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380186+0000","last_change":"2024-04-23T21:22:34.187783+0000","last_active":"2024-04-23T21:22:37.380186+0000","last_peered":"2024-04-23T21:22:37.380186+0000","last_clean":"2024-04-23T21:22:37.380186+0000","last_became_active":"2024-04-23T21:22:34.185924+0000","last_became_peered":"2024-04-23T21:22:34.185924+0000","last_unstale":"2024-04-23T21:22:37.380186+0000","last_undegraded":"2024-04-23T21:22:37.380186+0000","last_fullsized":"2024-04-23T21:22:37.380186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:22:23.821959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":44822653,"num_objects":67,"num_object_clones":14,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":35,"num_read":4926,"num_read_kb":14456,"num_write":1053,"num_write_kb":9858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"92'635","reported_seq":5403,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380301+0000","last_change":"2024-04-23T21:22:33.190596+0000","last_active":"2024-04-23T21:22:37.380301+0000","last_peered":"2024-04-23T21:22:37.380301+0000","last_clean":"2024-04-23T21:22:37.380301+0000","last_became_active":"2024-04-23T21:22:33.189427+0000","last_became_peered":"2024-04-23T21:22:33.189427+0000","last_unstale":"2024-04-23T21:22:37.380301+0000","last_undegraded":"2024-04-23T21:22:37.380301+0000","last_fullsized":"2024-04-23T21:22:37.380301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:31.040341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":44931716,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":37,"num_read":7690,"num_read_kb":24842,"num_write":813,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":5309940,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"93'499","reported_seq":2648,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387837+0000","last_change":"2024-04-23T21:22:38.387987+0000","last_active":"2024-04-23T21:22:38.387837+0000","last_peered":"2024-04-23T21:22:38.387837+0000","last_clean":"2024-04-23T21:22:38.387837+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:38.387837+0000","last_undegraded":"2024-04-23T21:22:38.387837+0000","last_fullsized":"2024-04-23T21:22:38.387837+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:54:49.539746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086574500000000001,"stat_sum":{"num_bytes":59572632,"num_objects":74,"num_object_clones":13,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":3820,"num_read_kb":30740,"num_write":539,"num_write_kb":28205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.16","version":"92'394","reported_seq":5056,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.476685+0000","last_change":"2024-04-23T21:22:38.387802+0000","last_active":"2024-04-23T21:22:38.476685+0000","last_peered":"2024-04-23T21:22:38.476685+0000","last_clean":"2024-04-23T21:22:38.476685+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:38.476685+0000","last_undegraded":"2024-04-23T21:22:38.476685+0000","last_fullsized":"2024-04-23T21:22:38.476685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:25.134906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087084800000000002,"stat_sum":{"num_bytes":61828228,"num_objects":55,"num_object_clones":11,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5990,"num_read_kb":36089,"num_write":431,"num_write_kb":7286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.15","version":"92'333","reported_seq":4406,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384339+0000","last_change":"2024-04-23T21:22:38.384573+0000","last_active":"2024-04-23T21:22:38.384339+0000","last_peered":"2024-04-23T21:22:38.384339+0000","last_clean":"2024-04-23T21:22:38.384339+0000","last_became_active":"2024-04-23T21:22:33.191679+0000","last_became_peered":"2024-04-23T21:22:33.191679+0000","last_unstale":"2024-04-23T21:22:38.384339+0000","last_undegraded":"2024-04-23T21:22:38.384339+0000","last_fullsized":"2024-04-23T21:22:38.384339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:05.849253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001744839,"stat_sum":{"num_bytes":74091838,"num_objects":72,"num_object_clones":18,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":33,"num_read":4322,"num_read_kb":26762,"num_write":336,"num_write_kb":9614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":5579029,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.14","version":"92'592","reported_seq":5561,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.390319+0000","last_change":"2024-04-23T21:22:38.394763+0000","last_active":"2024-04-23T21:22:38.390319+0000","last_peered":"2024-04-23T21:22:38.390319+0000","last_clean":"2024-04-23T21:22:38.390319+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:38.390319+0000","last_undegraded":"2024-04-23T21:22:38.390319+0000","last_fullsized":"2024-04-23T21:22:38.390319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:05:13.327032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012237974,"stat_sum":{"num_bytes":110248973,"num_objects":66,"num_object_clones":13,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":7406,"num_read_kb":52041,"num_write":742,"num_write_kb":17070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.438285+0000","last_change":"2024-04-23T21:22:30.359764+0000","last_active":"2024-04-23T21:22:38.438285+0000","last_peered":"2024-04-23T21:22:38.438285+0000","last_clean":"2024-04-23T21:22:38.438285+0000","last_became_active":"2024-04-23T21:22:30.358837+0000","last_became_peered":"2024-04-23T21:22:30.358837+0000","last_unstale":"2024-04-23T21:22:38.438285+0000","last_undegraded":"2024-04-23T21:22:38.438285+0000","last_fullsized":"2024-04-23T21:22:38.438285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:38.092515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"93'401","reported_seq":2610,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.466632+0000","last_change":"2024-04-23T21:22:38.387701+0000","last_active":"2024-04-23T21:22:38.466632+0000","last_peered":"2024-04-23T21:22:38.466632+0000","last_clean":"2024-04-23T21:22:38.466632+0000","last_became_active":"2024-04-23T21:22:33.186422+0000","last_became_peered":"2024-04-23T21:22:33.186422+0000","last_unstale":"2024-04-23T21:22:38.466632+0000","last_undegraded":"2024-04-23T21:22:38.466632+0000","last_fullsized":"2024-04-23T21:22:38.466632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:36.279822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578999999999997,"stat_sum":{"num_bytes":62894375,"num_objects":70,"num_object_clones":17,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":26,"num_read":2607,"num_read_kb":19018,"num_write":480,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":10723491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382567+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:38.382567+0000","last_peered":"2024-04-23T21:22:38.382567+0000","last_clean":"2024-04-23T21:22:38.382567+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:38.382567+0000","last_undegraded":"2024-04-23T21:22:38.382567+0000","last_fullsized":"2024-04-23T21:22:38.382567+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"93'351","reported_seq":3853,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.939487+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:37.939487+0000","last_peered":"2024-04-23T21:22:37.939487+0000","last_clean":"2024-04-23T21:22:37.939487+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:37.939487+0000","last_undegraded":"2024-04-23T21:22:37.939487+0000","last_fullsized":"2024-04-23T21:22:37.939487+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:07.398443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":86833958,"num_objects":62,"num_object_clones":17,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":26,"num_read":4697,"num_read_kb":34568,"num_write":316,"num_write_kb":17871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382318+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:38.382318+0000","last_peered":"2024-04-23T21:22:38.382318+0000","last_clean":"2024-04-23T21:22:38.382318+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:38.382318+0000","last_undegraded":"2024-04-23T21:22:38.382318+0000","last_fullsized":"2024-04-23T21:22:38.382318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"93'360","reported_seq":3403,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384073+0000","last_change":"2024-04-23T21:22:38.384310+0000","last_active":"2024-04-23T21:22:38.384073+0000","last_peered":"2024-04-23T21:22:38.384073+0000","last_clean":"2024-04-23T21:22:38.384073+0000","last_became_active":"2024-04-23T21:22:33.191620+0000","last_became_peered":"2024-04-23T21:22:33.191620+0000","last_unstale":"2024-04-23T21:22:38.384073+0000","last_undegraded":"2024-04-23T21:22:38.384073+0000","last_fullsized":"2024-04-23T21:22:38.384073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:04:04.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00161089,"stat_sum":{"num_bytes":61310486,"num_objects":66,"num_object_clones":11,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":3589,"num_read_kb":24366,"num_write":435,"num_write_kb":11827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":9761436,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386938+0000","last_change":"2024-04-23T21:22:30.359345+0000","last_active":"2024-04-23T21:22:38.386938+0000","last_peered":"2024-04-23T21:22:38.386938+0000","last_clean":"2024-04-23T21:22:38.386938+0000","last_became_active":"2024-04-23T21:22:30.359070+0000","last_became_peered":"2024-04-23T21:22:30.359070+0000","last_unstale":"2024-04-23T21:22:38.386938+0000","last_undegraded":"2024-04-23T21:22:38.386938+0000","last_fullsized":"2024-04-23T21:22:38.386938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:52.085200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"92'308","reported_seq":4025,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380217+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:37.380217+0000","last_peered":"2024-04-23T21:22:37.380217+0000","last_clean":"2024-04-23T21:22:37.380217+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:37.380217+0000","last_undegraded":"2024-04-23T21:22:37.380217+0000","last_fullsized":"2024-04-23T21:22:37.380217+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:27.966227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":90459570,"num_objects":79,"num_object_clones":19,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":40,"num_read":5935,"num_read_kb":33564,"num_write":327,"num_write_kb":18238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"79'345","reported_seq":8300,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382015+0000","last_change":"2024-04-23T21:22:38.382561+0000","last_active":"2024-04-23T21:22:38.382015+0000","last_peered":"2024-04-23T21:22:38.382015+0000","last_clean":"2024-04-23T21:22:38.382015+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:38.382015+0000","last_undegraded":"2024-04-23T21:22:38.382015+0000","last_fullsized":"2024-04-23T21:22:38.382015+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:31:27.059067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074128699999999998,"stat_sum":{"num_bytes":149341517,"num_objects":79,"num_object_clones":20,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":32,"num_read":8275,"num_read_kb":76033,"num_write":455,"num_write_kb":35588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":323,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382260+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:38.382260+0000","last_peered":"2024-04-23T21:22:38.382260+0000","last_clean":"2024-04-23T21:22:38.382260+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:38.382260+0000","last_undegraded":"2024-04-23T21:22:38.382260+0000","last_fullsized":"2024-04-23T21:22:38.382260+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380159+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:37.380159+0000","last_peered":"2024-04-23T21:22:37.380159+0000","last_clean":"2024-04-23T21:22:37.380159+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:37.380159+0000","last_undegraded":"2024-04-23T21:22:37.380159+0000","last_fullsized":"2024-04-23T21:22:37.380159+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'287","reported_seq":915,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.896604+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:37.896604+0000","last_peered":"2024-04-23T21:22:37.896604+0000","last_clean":"2024-04-23T21:22:37.896604+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:37.896604+0000","last_undegraded":"2024-04-23T21:22:37.896604+0000","last_fullsized":"2024-04-23T21:22:37.896604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":32288121,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":30,"num_read":3345,"num_read_kb":9678,"num_write":326,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382515+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:38.382515+0000","last_peered":"2024-04-23T21:22:38.382515+0000","last_clean":"2024-04-23T21:22:38.382515+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:38.382515+0000","last_undegraded":"2024-04-23T21:22:38.382515+0000","last_fullsized":"2024-04-23T21:22:38.382515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'651","reported_seq":7711,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483868+0000","last_change":"2024-04-23T21:22:38.395147+0000","last_active":"2024-04-23T21:22:38.483868+0000","last_peered":"2024-04-23T21:22:38.483868+0000","last_clean":"2024-04-23T21:22:38.483868+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:38.483868+0000","last_undegraded":"2024-04-23T21:22:38.483868+0000","last_fullsized":"2024-04-23T21:22:38.483868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:24.395971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359706,"stat_sum":{"num_bytes":88289494,"num_objects":85,"num_object_clones":22,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":34,"num_read":9955,"num_read_kb":57479,"num_write":910,"num_write_kb":33390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386737+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:38.386737+0000","last_peered":"2024-04-23T21:22:38.386737+0000","last_clean":"2024-04-23T21:22:38.386737+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:38.386737+0000","last_undegraded":"2024-04-23T21:22:38.386737+0000","last_fullsized":"2024-04-23T21:22:38.386737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'515","reported_seq":4728,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387255+0000","last_change":"2024-04-23T21:22:38.387429+0000","last_active":"2024-04-23T21:22:38.387255+0000","last_peered":"2024-04-23T21:22:38.387255+0000","last_clean":"2024-04-23T21:22:38.387255+0000","last_became_active":"2024-04-23T21:22:33.185729+0000","last_became_peered":"2024-04-23T21:22:33.185729+0000","last_unstale":"2024-04-23T21:22:38.387255+0000","last_undegraded":"2024-04-23T21:22:38.387255+0000","last_fullsized":"2024-04-23T21:22:38.387255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:17.916282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058378599999999998,"stat_sum":{"num_bytes":59185175,"num_objects":52,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":24,"num_read":6960,"num_read_kb":14315,"num_write":677,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382362+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:38.382362+0000","last_peered":"2024-04-23T21:22:38.382362+0000","last_clean":"2024-04-23T21:22:38.382362+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:38.382362+0000","last_undegraded":"2024-04-23T21:22:38.382362+0000","last_fullsized":"2024-04-23T21:22:38.382362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"93'241","reported_seq":2490,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.602023+0000","last_change":"2024-04-23T21:22:33.191726+0000","last_active":"2024-04-23T21:22:37.602023+0000","last_peered":"2024-04-23T21:22:37.602023+0000","last_clean":"2024-04-23T21:22:37.602023+0000","last_became_active":"2024-04-23T21:22:33.190419+0000","last_became_peered":"2024-04-23T21:22:33.190419+0000","last_unstale":"2024-04-23T21:22:37.602023+0000","last_undegraded":"2024-04-23T21:22:37.602023+0000","last_fullsized":"2024-04-23T21:22:37.602023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:45.249930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":35042921,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":35,"num_read":2161,"num_read_kb":15419,"num_write":257,"num_write_kb":12006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":13465209,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'296","reported_seq":7195,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.494395+0000","last_change":"2024-04-23T21:22:38.389596+0000","last_active":"2024-04-23T21:22:38.494395+0000","last_peered":"2024-04-23T21:22:38.494395+0000","last_clean":"2024-04-23T21:22:38.494395+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:38.494395+0000","last_undegraded":"2024-04-23T21:22:38.494395+0000","last_fullsized":"2024-04-23T21:22:38.494395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:37.705357+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077740689999999998,"stat_sum":{"num_bytes":104400617,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":34,"num_read":7010,"num_read_kb":46714,"num_write":329,"num_write_kb":15720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.a","version":"94'460","reported_seq":3385,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440408+0000","last_change":"2024-04-23T21:22:38.440408+0000","last_active":"2024-04-23T21:22:38.440408+0000","last_peered":"2024-04-23T21:22:38.440408+0000","last_clean":"2024-04-23T21:22:38.440408+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:38.440408+0000","last_undegraded":"2024-04-23T21:22:38.440408+0000","last_fullsized":"2024-04-23T21:22:38.440408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:03.627859+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001926042,"stat_sum":{"num_bytes":41021335,"num_objects":59,"num_object_clones":10,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":28,"num_read":5295,"num_read_kb":12681,"num_write":457,"num_write_kb":14975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.b","version":"94'402","reported_seq":2272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496515+0000","last_change":"2024-04-23T21:22:38.396815+0000","last_active":"2024-04-23T21:22:38.496515+0000","last_peered":"2024-04-23T21:22:38.496515+0000","last_clean":"2024-04-23T21:22:38.496515+0000","last_became_active":"2024-04-23T21:22:33.192667+0000","last_became_peered":"2024-04-23T21:22:33.192667+0000","last_unstale":"2024-04-23T21:22:38.496515+0000","last_undegraded":"2024-04-23T21:22:38.496515+0000","last_fullsized":"2024-04-23T21:22:38.496515+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:24.257165+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.014784952000000001,"stat_sum":{"num_bytes":64284813,"num_objects":66,"num_object_clones":14,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":30,"num_read":2590,"num_read_kb":25806,"num_write":435,"num_write_kb":14302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":9997927,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.c","version":"92'351","reported_seq":1633,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387010+0000","last_change":"2024-04-23T21:22:38.387207+0000","last_active":"2024-04-23T21:22:38.387010+0000","last_peered":"2024-04-23T21:22:38.387010+0000","last_clean":"2024-04-23T21:22:38.387010+0000","last_became_active":"2024-04-23T21:22:33.185774+0000","last_became_peered":"2024-04-23T21:22:33.185774+0000","last_unstale":"2024-04-23T21:22:38.387010+0000","last_undegraded":"2024-04-23T21:22:38.387010+0000","last_fullsized":"2024-04-23T21:22:38.387010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:50:58.291001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000369157,"stat_sum":{"num_bytes":95852508,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":34,"num_read":1441,"num_read_kb":18659,"num_write":330,"num_write_kb":17730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.d","version":"94'264","reported_seq":2543,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389126+0000","last_change":"2024-04-23T21:22:38.389428+0000","last_active":"2024-04-23T21:22:38.389126+0000","last_peered":"2024-04-23T21:22:38.389126+0000","last_clean":"2024-04-23T21:22:38.389126+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:38.389126+0000","last_undegraded":"2024-04-23T21:22:38.389126+0000","last_fullsized":"2024-04-23T21:22:38.389126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:54:15.435558+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0063591359999999996,"stat_sum":{"num_bytes":45966183,"num_objects":54,"num_object_clones":9,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":29,"num_read":2501,"num_read_kb":11452,"num_write":248,"num_write_kb":11068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.e","version":"94'506","reported_seq":1920,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496079+0000","last_change":"2024-04-23T21:22:38.384963+0000","last_active":"2024-04-23T21:22:38.496079+0000","last_peered":"2024-04-23T21:22:38.496079+0000","last_clean":"2024-04-23T21:22:38.496079+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:38.496079+0000","last_undegraded":"2024-04-23T21:22:38.496079+0000","last_fullsized":"2024-04-23T21:22:38.496079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:09:46.962009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320499999999999,"stat_sum":{"num_bytes":48671382,"num_objects":55,"num_object_clones":10,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":27,"num_read":3448,"num_read_kb":11204,"num_write":630,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.f","version":"94'896","reported_seq":9954,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.395078+0000","last_change":"2024-04-23T21:22:38.395242+0000","last_active":"2024-04-23T21:22:38.395078+0000","last_peered":"2024-04-23T21:22:38.395078+0000","last_clean":"2024-04-23T21:22:38.395078+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:38.395078+0000","last_undegraded":"2024-04-23T21:22:38.395078+0000","last_fullsized":"2024-04-23T21:22:38.395078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:41:01.685567+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012161335000000001,"stat_sum":{"num_bytes":127130709,"num_objects":81,"num_object_clones":18,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":32,"num_read":13152,"num_read_kb":58727,"num_write":1272,"num_write_kb":16409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.10","version":"93'414","reported_seq":3099,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483088+0000","last_change":"2024-04-23T21:22:38.438886+0000","last_active":"2024-04-23T21:22:38.483088+0000","last_peered":"2024-04-23T21:22:38.483088+0000","last_clean":"2024-04-23T21:22:38.483088+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:38.483088+0000","last_undegraded":"2024-04-23T21:22:38.483088+0000","last_fullsized":"2024-04-23T21:22:38.483088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:47:55.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047995299999999998,"stat_sum":{"num_bytes":101761897,"num_objects":74,"num_object_clones":17,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":30,"num_read":4138,"num_read_kb":33523,"num_write":481,"num_write_kb":20694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.11","version":"94'253","reported_seq":2188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389732+0000","last_change":"2024-04-23T21:22:38.390017+0000","last_active":"2024-04-23T21:22:38.389732+0000","last_peered":"2024-04-23T21:22:38.389732+0000","last_clean":"2024-04-23T21:22:38.389732+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:38.389732+0000","last_undegraded":"2024-04-23T21:22:38.389732+0000","last_fullsized":"2024-04-23T21:22:38.389732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:53:16.375004+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077344930000000003,"stat_sum":{"num_bytes":91411180,"num_objects":66,"num_object_clones":17,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":27,"num_read":1874,"num_read_kb":20591,"num_write":283,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.12","version":"94'789","reported_seq":2928,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.454378+0000","last_change":"2024-04-23T21:22:38.384028+0000","last_active":"2024-04-23T21:22:38.454378+0000","last_peered":"2024-04-23T21:22:38.454378+0000","last_clean":"2024-04-23T21:22:38.454378+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:38.454378+0000","last_undegraded":"2024-04-23T21:22:38.454378+0000","last_fullsized":"2024-04-23T21:22:38.454378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:26:45.125365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014285649999999999,"stat_sum":{"num_bytes":80276131,"num_objects":72,"num_object_clones":12,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":27,"num_read":5353,"num_read_kb":39496,"num_write":1086,"num_write_kb":21634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.13","version":"93'816","reported_seq":2742,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382845+0000","last_change":"2024-04-23T21:22:38.383310+0000","last_active":"2024-04-23T21:22:38.382845+0000","last_peered":"2024-04-23T21:22:38.382845+0000","last_clean":"2024-04-23T21:22:38.382845+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:38.382845+0000","last_undegraded":"2024-04-23T21:22:38.382845+0000","last_fullsized":"2024-04-23T21:22:38.382845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:11:00.823606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084378100000000004,"stat_sum":{"num_bytes":83105146,"num_objects":70,"num_object_clones":14,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":5188,"num_read_kb":39132,"num_write":1248,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2385400573,"num_objects":2175,"num_object_clones":497,"num_object_copies":4350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":1004,"num_read":177040,"num_read_kb":1010729,"num_write":17863,"num_write_kb":597645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92680192,"data_stored":927288416,"data_compressed":55633033,"data_compressed_allocated":61050880,"data_compressed_original":899446117,"omap_allocated":0,"internal_metadata":0},"log_size":14685,"ondisk_log_size":14685,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65288,"kb_used_data":16060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94306552,"statfs":{"total":96636764160,"available":96569909248,"internally_reserved":0,"allocated":16445440,"data_stored":144462534,"data_compressed":9363404,"data_compressed_allocated":10027008,"data_compressed_original":139355245,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42648,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329192,"statfs":{"total":96636764160,"available":96593092608,"internally_reserved":0,"allocated":7475200,"data_stored":64982199,"data_compressed":3626218,"data_compressed_allocated":4145152,"data_compressed_original":62472853,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49644,"kb_used_data":10136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322196,"statfs":{"total":96636764160,"available":96585928704,"internally_reserved":0,"allocated":10379264,"data_stored":87504141,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83980,"kb_used_data":26436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94287860,"statfs":{"total":96636764160,"available":96550768640,"internally_reserved":0,"allocated":27070464,"data_stored":267140093,"data_compressed":15733233,"data_compressed_allocated":17158144,"data_compressed_original":258789727,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39128,"kb_used_data":7936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332712,"statfs":{"total":96636764160,"available":96596697088,"internally_reserved":0,"allocated":8126464,"data_stored":65599087,"data_compressed":3837804,"data_compressed_allocated":4386816,"data_compressed_original":62678344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528331,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46560,"kb_used_data":9876,"kb_used_omap":6,"kb_used_meta":36665,"kb_avail":94325280,"statfs":{"total":96636764160,"available":96589086720,"internally_reserved":0,"allocated":10113024,"data_stored":97825612,"data_compressed":6089726,"data_compressed_allocated":6176768,"data_compressed_original":94863024,"omap_allocated":6748,"internal_metadata":37545380},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69020,"kb_used_data":19792,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302820,"statfs":{"total":96636764160,"available":96566087680,"internally_reserved":0,"allocated":20267008,"data_stored":199150039,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19156992,"data_stored":198632385,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9138176,"data_stored":96862759,"data_compressed":6055668,"data_compressed_allocated":6131712,"data_compressed_original":94273200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":64762460,"data_compressed":3818707,"data_compressed_allocated":4358144,"data_compressed_original":62364230,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":6676463,"data_compressed":354993,"data_compressed_allocated":483328,"data_compressed_original":6245965,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":266049296,"data_compressed":15712561,"data_compressed_allocated":17125376,"data_compressed_original":258392415,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9117696,"data_stored":86851690,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6336512,"data_stored":64152842,"data_compressed":3607499,"data_compressed_allocated":4116480,"data_compressed_original":62166931,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15212544,"data_stored":143300521,"data_compressed":9330982,"data_compressed_allocated":9981952,"data_compressed_original":138781805,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:42.093 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-23T21:22:42.093 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:42.093 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-23T21:22:42.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:22:42.165 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:42.165+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:42.183 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:42.183 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:42.183 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:42.183 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:42.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:42.266 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:42.267+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:42.298 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:42.298 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:42.299 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:42.299 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:42.299 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:42.308 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:42.392 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:42.393+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 98, 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:22:42.435 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:22:42.409411+0000", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:21:48.929598+0000", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:22:29.344375+0000", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5801984, 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "noscrub", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 5, 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:22:42.436 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "luminous", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:22:42.437 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:22:42.438 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:42.439 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:42.440 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:42.441 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 2.9900000095367432, 2024-04-23T21:22:42.442 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 2.9900000095367432, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.87999999523162842, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2.630000114440918, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2.630000114440918, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:42.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:22:42.444 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:42.445 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:42.446 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:42.447 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 8, 2024-04-23T21:22:42.448 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 8, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.25, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:22:42.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:22:42.450 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "98", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:42.451 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 51, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 98, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:42.452 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:42.453 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:42.454 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 1.9900000095367432, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 1.9900000095367432, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.87999999523162842, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.75, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.75, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:22:42.455 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.04681396484375, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 87, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.456 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.457 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:22:42.458 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:22:42.459 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.460 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 13, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 17, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 45, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 89, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:22:42.461 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.462 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/723341542", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/723341542", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/723341542", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/723341542", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:22:42.466 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:22:42.467 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.471 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 0, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 0, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.472 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.473 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.474 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.475 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:22:42.476 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.477 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.480 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.481 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.486 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:22:42.487 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.488 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.489 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.490 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.491 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.492 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.493 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:42.494 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:22:42.497 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:42.498 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.500 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 65536, 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:42.502 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.503 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.504 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [ 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.6", 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "mappings": [ 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "from": 6, 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "to": 7 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [], 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:22:42.505 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [ 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.506 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 47, 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 5 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [ 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 47, 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 1 2024-04-23T21:22:42.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [], 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.508 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:42.511 INFO:tasks.thrashosds.thrasher:j is {'epoch': 98, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:22:42.409411+0000', 'last_up_change': '2024-04-23T21:21:48.929598+0000', 'last_in_change': '2024-04-23T21:22:29.344375+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 51, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.04681396484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 87, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 17, 'up_from': 45, 'up_thru': 89, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 723341542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 723341542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 723341542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 723341542}]}, 'public_addr': '172.21.15.111:6817/723341542', 'cluster_addr': '172.21.15.111:6819/723341542', 'heartbeat_back_addr': '172.21.15.111:6823/723341542', 'heartbeat_front_addr': '172.21.15.111:6821/723341542', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.6', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 47, 'length': 5}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 47, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:22:42.511 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:42.512 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:42.512 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:42.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:42.512 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.819789376999999, 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:42.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:42.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:42.516 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.819757269999997, 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.517 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.819733884000001, 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.521 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:42.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:42.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.819713577000002, 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.529 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.819693764999997, 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818622367000003, 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:42.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:42.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818589021999998, 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:42.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:42.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818557493, 2024-04-23T21:22:42.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.546 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818530103999997, 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:42.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:42.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:42.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818518198, 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.554 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818499954000004, 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.555 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:42.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:42.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:42.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818490136999998, 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.559 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818479889999999, 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.563 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:42.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:42.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818470239, 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:42.567 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.571 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.818436740000003, 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:42.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.814811315999997, 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:42.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:42.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.814793447, 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.580 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.814774718000002, 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.584 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:42.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.814756289999998, 2024-04-23T21:22:42.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:42.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:42.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.592 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.814738384999998, 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:42.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.596 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:42.597 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:42.597 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:42.597 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:42.598 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:42.510+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:42.641 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:42.642+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:42.701 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:42.702 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:42.702 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:42.702 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:42.710 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:42.711 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:42.713+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:42.812 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:42.812 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:42.813 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:42.813 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:42.813 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:42.813 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:42.813+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:42.821 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:42.881 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:42.894 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-04-23T21:22:42.414626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2385990941,"num_objects":2177,"num_object_clones":497,"num_object_copies":4354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2177,"num_whiteouts":1004,"num_read":177156,"num_read_kb":1010829,"num_write":18022,"num_write_kb":600061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14802,"ondisk_log_size":14802,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":396268,"kb_used_data":97536,"kb_used_omap":15,"kb_used_meta":298608,"kb_avail":660206612,"statfs":{"total":676457349120,"available":676051570688,"internally_reserved":0,"allocated":99876864,"data_stored":926663705,"data_compressed":55403008,"data_compressed_allocated":60747776,"data_compressed_original":895380764,"omap_allocated":16311,"internal_metadata":305774665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":77530590,"num_objects":46,"num_object_clones":17,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":25,"num_read":4102,"num_read_kb":9950,"num_write":565,"num_write_kb":25366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033515"},"pg_stats":[{"pgid":"3.1f","version":"92'418","reported_seq":4067,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387746+0000","last_change":"2024-04-23T21:22:38.387900+0000","last_active":"2024-04-23T21:22:38.387746+0000","last_peered":"2024-04-23T21:22:38.387746+0000","last_clean":"2024-04-23T21:22:38.387746+0000","last_became_active":"2024-04-23T21:21:32.840704+0000","last_became_peered":"2024-04-23T21:21:32.840704+0000","last_unstale":"2024-04-23T21:22:38.387746+0000","last_undegraded":"2024-04-23T21:22:38.387746+0000","last_fullsized":"2024-04-23T21:22:38.387746+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:46:08.750064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000807083,"stat_sum":{"num_bytes":73070064,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5911,"num_read_kb":16706,"num_write":482,"num_write_kb":33855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1e","version":"94'374","reported_seq":6612,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440046+0000","last_change":"2024-04-23T21:22:38.440210+0000","last_active":"2024-04-23T21:22:38.440046+0000","last_peered":"2024-04-23T21:22:38.440046+0000","last_clean":"2024-04-23T21:22:38.440046+0000","last_became_active":"2024-04-23T21:22:33.186559+0000","last_became_peered":"2024-04-23T21:22:33.186559+0000","last_unstale":"2024-04-23T21:22:38.440046+0000","last_undegraded":"2024-04-23T21:22:38.440046+0000","last_fullsized":"2024-04-23T21:22:38.440046+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:32:32.760937+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0018364259999999999,"stat_sum":{"num_bytes":76868783,"num_objects":72,"num_object_clones":19,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":32,"num_read":6375,"num_read_kb":47205,"num_write":430,"num_write_kb":24897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1d","version":"83'279","reported_seq":8176,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387401+0000","last_change":"2024-04-23T21:22:38.387584+0000","last_active":"2024-04-23T21:22:38.387401+0000","last_peered":"2024-04-23T21:22:38.387401+0000","last_clean":"2024-04-23T21:22:38.387401+0000","last_became_active":"2024-04-23T21:21:17.780284+0000","last_became_peered":"2024-04-23T21:21:17.780284+0000","last_unstale":"2024-04-23T21:22:38.387401+0000","last_undegraded":"2024-04-23T21:22:38.387401+0000","last_fullsized":"2024-04-23T21:22:38.387401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:42:32.941280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070416999999999995,"stat_sum":{"num_bytes":46779874,"num_objects":65,"num_object_clones":14,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":32,"num_read":8295,"num_read_kb":31836,"num_write":262,"num_write_kb":7752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1c","version":"94'397","reported_seq":2921,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.473422+0000","last_change":"2024-04-23T21:22:38.387273+0000","last_active":"2024-04-23T21:22:38.473422+0000","last_peered":"2024-04-23T21:22:38.473422+0000","last_clean":"2024-04-23T21:22:38.473422+0000","last_became_active":"2024-04-23T21:21:17.782211+0000","last_became_peered":"2024-04-23T21:21:17.782211+0000","last_unstale":"2024-04-23T21:22:38.473422+0000","last_undegraded":"2024-04-23T21:22:38.473422+0000","last_fullsized":"2024-04-23T21:22:38.473422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:56:35.225413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000435031,"stat_sum":{"num_bytes":69525595,"num_objects":79,"num_object_clones":21,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":39,"num_read":2897,"num_read_kb":24662,"num_write":465,"num_write_kb":16868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1b","version":"91'327","reported_seq":9994,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389864+0000","last_change":"2024-04-23T21:22:38.390268+0000","last_active":"2024-04-23T21:22:38.389864+0000","last_peered":"2024-04-23T21:22:38.389864+0000","last_clean":"2024-04-23T21:22:38.389864+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:38.389864+0000","last_undegraded":"2024-04-23T21:22:38.389864+0000","last_fullsized":"2024-04-23T21:22:38.389864+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:11.320083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080390080000000003,"stat_sum":{"num_bytes":114321291,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":31,"num_read":10911,"num_read_kb":51379,"num_write":339,"num_write_kb":56482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.1a","version":"94'781","reported_seq":5484,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.482706+0000","last_change":"2024-04-23T21:22:38.389806+0000","last_active":"2024-04-23T21:22:38.482706+0000","last_peered":"2024-04-23T21:22:38.482706+0000","last_clean":"2024-04-23T21:22:38.482706+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:38.482706+0000","last_undegraded":"2024-04-23T21:22:38.482706+0000","last_fullsized":"2024-04-23T21:22:38.482706+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:44.982850+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0079933109999999995,"stat_sum":{"num_bytes":59811408,"num_objects":61,"num_object_clones":11,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":29,"num_read":8983,"num_read_kb":51586,"num_write":989,"num_write_kb":37904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.19","version":"91'750","reported_seq":2570,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380186+0000","last_change":"2024-04-23T21:22:34.187783+0000","last_active":"2024-04-23T21:22:37.380186+0000","last_peered":"2024-04-23T21:22:37.380186+0000","last_clean":"2024-04-23T21:22:37.380186+0000","last_became_active":"2024-04-23T21:22:34.185924+0000","last_became_peered":"2024-04-23T21:22:34.185924+0000","last_unstale":"2024-04-23T21:22:37.380186+0000","last_undegraded":"2024-04-23T21:22:37.380186+0000","last_fullsized":"2024-04-23T21:22:37.380186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":750,"log_dups_size":0,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:22:23.821959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065535300000000001,"stat_sum":{"num_bytes":44822653,"num_objects":67,"num_object_clones":14,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":35,"num_read":4926,"num_read_kb":14456,"num_write":1053,"num_write_kb":9858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"92'635","reported_seq":5403,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380301+0000","last_change":"2024-04-23T21:22:33.190596+0000","last_active":"2024-04-23T21:22:37.380301+0000","last_peered":"2024-04-23T21:22:37.380301+0000","last_clean":"2024-04-23T21:22:37.380301+0000","last_became_active":"2024-04-23T21:22:33.189427+0000","last_became_peered":"2024-04-23T21:22:33.189427+0000","last_unstale":"2024-04-23T21:22:37.380301+0000","last_undegraded":"2024-04-23T21:22:37.380301+0000","last_fullsized":"2024-04-23T21:22:37.380301+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:07:31.040341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058483199999999997,"stat_sum":{"num_bytes":44931716,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":37,"num_read":7690,"num_read_kb":24842,"num_write":813,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":5309940,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"93'499","reported_seq":2648,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387837+0000","last_change":"2024-04-23T21:22:38.387987+0000","last_active":"2024-04-23T21:22:38.387837+0000","last_peered":"2024-04-23T21:22:38.387837+0000","last_clean":"2024-04-23T21:22:38.387837+0000","last_became_active":"2024-04-23T21:21:32.841059+0000","last_became_peered":"2024-04-23T21:21:32.841059+0000","last_unstale":"2024-04-23T21:22:38.387837+0000","last_undegraded":"2024-04-23T21:22:38.387837+0000","last_fullsized":"2024-04-23T21:22:38.387837+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:54:49.539746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086574500000000001,"stat_sum":{"num_bytes":59572632,"num_objects":74,"num_object_clones":13,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":3820,"num_read_kb":30740,"num_write":539,"num_write_kb":28205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.16","version":"92'394","reported_seq":5056,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.476685+0000","last_change":"2024-04-23T21:22:38.387802+0000","last_active":"2024-04-23T21:22:38.476685+0000","last_peered":"2024-04-23T21:22:38.476685+0000","last_clean":"2024-04-23T21:22:38.476685+0000","last_became_active":"2024-04-23T21:21:17.780114+0000","last_became_peered":"2024-04-23T21:21:17.780114+0000","last_unstale":"2024-04-23T21:22:38.476685+0000","last_undegraded":"2024-04-23T21:22:38.476685+0000","last_fullsized":"2024-04-23T21:22:38.476685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:25.134906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087084800000000002,"stat_sum":{"num_bytes":61828228,"num_objects":55,"num_object_clones":11,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":5990,"num_read_kb":36089,"num_write":431,"num_write_kb":7286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.15","version":"92'333","reported_seq":4406,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384339+0000","last_change":"2024-04-23T21:22:38.384573+0000","last_active":"2024-04-23T21:22:38.384339+0000","last_peered":"2024-04-23T21:22:38.384339+0000","last_clean":"2024-04-23T21:22:38.384339+0000","last_became_active":"2024-04-23T21:22:33.191679+0000","last_became_peered":"2024-04-23T21:22:33.191679+0000","last_unstale":"2024-04-23T21:22:38.384339+0000","last_undegraded":"2024-04-23T21:22:38.384339+0000","last_fullsized":"2024-04-23T21:22:38.384339+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:19:05.849253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001744839,"stat_sum":{"num_bytes":74091838,"num_objects":72,"num_object_clones":18,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":33,"num_read":4322,"num_read_kb":26762,"num_write":336,"num_write_kb":9614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":5579029,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.14","version":"92'592","reported_seq":5561,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.390319+0000","last_change":"2024-04-23T21:22:38.394763+0000","last_active":"2024-04-23T21:22:38.390319+0000","last_peered":"2024-04-23T21:22:38.390319+0000","last_clean":"2024-04-23T21:22:38.390319+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:38.390319+0000","last_undegraded":"2024-04-23T21:22:38.390319+0000","last_fullsized":"2024-04-23T21:22:38.390319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:05:13.327032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012237974,"stat_sum":{"num_bytes":110248973,"num_objects":66,"num_object_clones":13,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":7406,"num_read_kb":52041,"num_write":742,"num_write_kb":17070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":173,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.438285+0000","last_change":"2024-04-23T21:22:30.359764+0000","last_active":"2024-04-23T21:22:38.438285+0000","last_peered":"2024-04-23T21:22:38.438285+0000","last_clean":"2024-04-23T21:22:38.438285+0000","last_became_active":"2024-04-23T21:22:30.358837+0000","last_became_peered":"2024-04-23T21:22:30.358837+0000","last_unstale":"2024-04-23T21:22:38.438285+0000","last_undegraded":"2024-04-23T21:22:38.438285+0000","last_fullsized":"2024-04-23T21:22:38.438285+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:38.092515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"93'401","reported_seq":2610,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.466632+0000","last_change":"2024-04-23T21:22:38.387701+0000","last_active":"2024-04-23T21:22:38.466632+0000","last_peered":"2024-04-23T21:22:38.466632+0000","last_clean":"2024-04-23T21:22:38.466632+0000","last_became_active":"2024-04-23T21:22:33.186422+0000","last_became_peered":"2024-04-23T21:22:33.186422+0000","last_unstale":"2024-04-23T21:22:38.466632+0000","last_undegraded":"2024-04-23T21:22:38.466632+0000","last_fullsized":"2024-04-23T21:22:38.466632+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":401,"log_dups_size":0,"ondisk_log_size":401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:50:36.279822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578999999999997,"stat_sum":{"num_bytes":62894375,"num_objects":70,"num_object_clones":17,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":26,"num_read":2607,"num_read_kb":19018,"num_write":480,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":10723491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382567+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:38.382567+0000","last_peered":"2024-04-23T21:22:38.382567+0000","last_clean":"2024-04-23T21:22:38.382567+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:38.382567+0000","last_undegraded":"2024-04-23T21:22:38.382567+0000","last_fullsized":"2024-04-23T21:22:38.382567+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"93'351","reported_seq":3853,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.939487+0000","last_change":"2024-04-23T21:21:52.961326+0000","last_active":"2024-04-23T21:22:37.939487+0000","last_peered":"2024-04-23T21:22:37.939487+0000","last_clean":"2024-04-23T21:22:37.939487+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:37.939487+0000","last_undegraded":"2024-04-23T21:22:37.939487+0000","last_fullsized":"2024-04-23T21:22:37.939487+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:55:07.398443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091513399999999996,"stat_sum":{"num_bytes":86833958,"num_objects":62,"num_object_clones":17,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":26,"num_read":4697,"num_read_kb":34568,"num_write":316,"num_write_kb":17871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382318+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:38.382318+0000","last_peered":"2024-04-23T21:22:38.382318+0000","last_clean":"2024-04-23T21:22:38.382318+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:38.382318+0000","last_undegraded":"2024-04-23T21:22:38.382318+0000","last_fullsized":"2024-04-23T21:22:38.382318+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"93'360","reported_seq":3403,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.384073+0000","last_change":"2024-04-23T21:22:38.384310+0000","last_active":"2024-04-23T21:22:38.384073+0000","last_peered":"2024-04-23T21:22:38.384073+0000","last_clean":"2024-04-23T21:22:38.384073+0000","last_became_active":"2024-04-23T21:22:33.191620+0000","last_became_peered":"2024-04-23T21:22:33.191620+0000","last_unstale":"2024-04-23T21:22:38.384073+0000","last_undegraded":"2024-04-23T21:22:38.384073+0000","last_fullsized":"2024-04-23T21:22:38.384073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:04:04.227524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00161089,"stat_sum":{"num_bytes":61310486,"num_objects":66,"num_object_clones":11,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":31,"num_read":3589,"num_read_kb":24366,"num_write":435,"num_write_kb":11827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":9761436,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386938+0000","last_change":"2024-04-23T21:22:30.359345+0000","last_active":"2024-04-23T21:22:38.386938+0000","last_peered":"2024-04-23T21:22:38.386938+0000","last_clean":"2024-04-23T21:22:38.386938+0000","last_became_active":"2024-04-23T21:22:30.359070+0000","last_became_peered":"2024-04-23T21:22:30.359070+0000","last_unstale":"2024-04-23T21:22:38.386938+0000","last_undegraded":"2024-04-23T21:22:38.386938+0000","last_fullsized":"2024-04-23T21:22:38.386938+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:56:52.085200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"92'308","reported_seq":4025,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380217+0000","last_change":"2024-04-23T21:21:50.015640+0000","last_active":"2024-04-23T21:22:37.380217+0000","last_peered":"2024-04-23T21:22:37.380217+0000","last_clean":"2024-04-23T21:22:37.380217+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:37.380217+0000","last_undegraded":"2024-04-23T21:22:37.380217+0000","last_fullsized":"2024-04-23T21:22:37.380217+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:47:27.966227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061388199999999995,"stat_sum":{"num_bytes":90459570,"num_objects":79,"num_object_clones":19,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":40,"num_read":5935,"num_read_kb":33564,"num_write":327,"num_write_kb":18238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"79'345","reported_seq":8300,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382015+0000","last_change":"2024-04-23T21:22:38.382561+0000","last_active":"2024-04-23T21:22:38.382015+0000","last_peered":"2024-04-23T21:22:38.382015+0000","last_clean":"2024-04-23T21:22:38.382015+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:38.382015+0000","last_undegraded":"2024-04-23T21:22:38.382015+0000","last_fullsized":"2024-04-23T21:22:38.382015+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:31:27.059067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074128699999999998,"stat_sum":{"num_bytes":149341517,"num_objects":79,"num_object_clones":20,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":32,"num_read":8275,"num_read_kb":76033,"num_write":455,"num_write_kb":35588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"2.0","version":"16'117","reported_seq":323,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382260+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:38.382260+0000","last_peered":"2024-04-23T21:22:38.382260+0000","last_clean":"2024-04-23T21:22:38.382260+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:38.382260+0000","last_undegraded":"2024-04-23T21:22:38.382260+0000","last_fullsized":"2024-04-23T21:22:38.382260+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":114,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.380159+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:37.380159+0000","last_peered":"2024-04-23T21:22:37.380159+0000","last_clean":"2024-04-23T21:22:37.380159+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:37.380159+0000","last_undegraded":"2024-04-23T21:22:37.380159+0000","last_fullsized":"2024-04-23T21:22:37.380159+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"93'287","reported_seq":915,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.896604+0000","last_change":"2024-04-23T21:21:49.981289+0000","last_active":"2024-04-23T21:22:37.896604+0000","last_peered":"2024-04-23T21:22:37.896604+0000","last_clean":"2024-04-23T21:22:37.896604+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:37.896604+0000","last_undegraded":"2024-04-23T21:22:37.896604+0000","last_fullsized":"2024-04-23T21:22:37.896604+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:57:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098719299999999992,"stat_sum":{"num_bytes":32288121,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":30,"num_read":3345,"num_read_kb":9678,"num_write":326,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382515+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:38.382515+0000","last_peered":"2024-04-23T21:22:38.382515+0000","last_clean":"2024-04-23T21:22:38.382515+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:38.382515+0000","last_undegraded":"2024-04-23T21:22:38.382515+0000","last_fullsized":"2024-04-23T21:22:38.382515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"94'651","reported_seq":7711,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483868+0000","last_change":"2024-04-23T21:22:38.395147+0000","last_active":"2024-04-23T21:22:38.483868+0000","last_peered":"2024-04-23T21:22:38.483868+0000","last_clean":"2024-04-23T21:22:38.483868+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:38.483868+0000","last_undegraded":"2024-04-23T21:22:38.483868+0000","last_fullsized":"2024-04-23T21:22:38.483868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:24.395971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012359706,"stat_sum":{"num_bytes":88289494,"num_objects":85,"num_object_clones":22,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":34,"num_read":9955,"num_read_kb":57479,"num_write":910,"num_write_kb":33390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.386737+0000","last_change":"2024-04-23T21:21:09.823708+0000","last_active":"2024-04-23T21:22:38.386737+0000","last_peered":"2024-04-23T21:22:38.386737+0000","last_clean":"2024-04-23T21:22:38.386737+0000","last_became_active":"2024-04-23T21:21:09.823391+0000","last_became_peered":"2024-04-23T21:21:09.823391+0000","last_unstale":"2024-04-23T21:22:38.386737+0000","last_undegraded":"2024-04-23T21:22:38.386737+0000","last_fullsized":"2024-04-23T21:22:38.386737+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:35:50.829535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"93'515","reported_seq":4728,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387255+0000","last_change":"2024-04-23T21:22:38.387429+0000","last_active":"2024-04-23T21:22:38.387255+0000","last_peered":"2024-04-23T21:22:38.387255+0000","last_clean":"2024-04-23T21:22:38.387255+0000","last_became_active":"2024-04-23T21:22:33.185729+0000","last_became_peered":"2024-04-23T21:22:33.185729+0000","last_unstale":"2024-04-23T21:22:38.387255+0000","last_undegraded":"2024-04-23T21:22:38.387255+0000","last_fullsized":"2024-04-23T21:22:38.387255+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:41:17.916282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058378599999999998,"stat_sum":{"num_bytes":59185175,"num_objects":52,"num_object_clones":10,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":24,"num_read":6960,"num_read_kb":14315,"num_write":677,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":168,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382362+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:38.382362+0000","last_peered":"2024-04-23T21:22:38.382362+0000","last_clean":"2024-04-23T21:22:38.382362+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:38.382362+0000","last_undegraded":"2024-04-23T21:22:38.382362+0000","last_fullsized":"2024-04-23T21:22:38.382362+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"93'241","reported_seq":2490,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-23T21:22:37.602023+0000","last_change":"2024-04-23T21:22:33.191726+0000","last_active":"2024-04-23T21:22:37.602023+0000","last_peered":"2024-04-23T21:22:37.602023+0000","last_clean":"2024-04-23T21:22:37.602023+0000","last_became_active":"2024-04-23T21:22:33.190419+0000","last_became_peered":"2024-04-23T21:22:33.190419+0000","last_unstale":"2024-04-23T21:22:37.602023+0000","last_undegraded":"2024-04-23T21:22:37.602023+0000","last_fullsized":"2024-04-23T21:22:37.602023+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:49:45.249930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063354100000000005,"stat_sum":{"num_bytes":35042921,"num_objects":70,"num_object_clones":20,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":35,"num_read":2161,"num_read_kb":15419,"num_write":257,"num_write_kb":12006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":13465209,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"94'296","reported_seq":7195,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.494395+0000","last_change":"2024-04-23T21:22:38.389596+0000","last_active":"2024-04-23T21:22:38.494395+0000","last_peered":"2024-04-23T21:22:38.494395+0000","last_clean":"2024-04-23T21:22:38.494395+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:38.494395+0000","last_undegraded":"2024-04-23T21:22:38.494395+0000","last_fullsized":"2024-04-23T21:22:38.494395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:15:37.705357+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077740689999999998,"stat_sum":{"num_bytes":104400617,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":34,"num_read":7010,"num_read_kb":46714,"num_write":329,"num_write_kb":15720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.a","version":"94'460","reported_seq":3385,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.440408+0000","last_change":"2024-04-23T21:22:38.440408+0000","last_active":"2024-04-23T21:22:38.440408+0000","last_peered":"2024-04-23T21:22:38.440408+0000","last_clean":"2024-04-23T21:22:38.440408+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:38.440408+0000","last_undegraded":"2024-04-23T21:22:38.440408+0000","last_fullsized":"2024-04-23T21:22:38.440408+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:03.627859+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001926042,"stat_sum":{"num_bytes":41021335,"num_objects":59,"num_object_clones":10,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":28,"num_read":5295,"num_read_kb":12681,"num_write":457,"num_write_kb":14975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.b","version":"94'402","reported_seq":2272,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496515+0000","last_change":"2024-04-23T21:22:38.396815+0000","last_active":"2024-04-23T21:22:38.496515+0000","last_peered":"2024-04-23T21:22:38.496515+0000","last_clean":"2024-04-23T21:22:38.496515+0000","last_became_active":"2024-04-23T21:22:33.192667+0000","last_became_peered":"2024-04-23T21:22:33.192667+0000","last_unstale":"2024-04-23T21:22:38.496515+0000","last_undegraded":"2024-04-23T21:22:38.496515+0000","last_fullsized":"2024-04-23T21:22:38.496515+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:52:24.257165+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.014784952000000001,"stat_sum":{"num_bytes":64284813,"num_objects":66,"num_object_clones":14,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":30,"num_read":2590,"num_read_kb":25806,"num_write":435,"num_write_kb":14302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":9997927,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.c","version":"92'351","reported_seq":1633,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.387010+0000","last_change":"2024-04-23T21:22:38.387207+0000","last_active":"2024-04-23T21:22:38.387010+0000","last_peered":"2024-04-23T21:22:38.387010+0000","last_clean":"2024-04-23T21:22:38.387010+0000","last_became_active":"2024-04-23T21:22:33.185774+0000","last_became_peered":"2024-04-23T21:22:33.185774+0000","last_unstale":"2024-04-23T21:22:38.387010+0000","last_undegraded":"2024-04-23T21:22:38.387010+0000","last_fullsized":"2024-04-23T21:22:38.387010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:50:58.291001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000369157,"stat_sum":{"num_bytes":95852508,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":34,"num_read":1441,"num_read_kb":18659,"num_write":330,"num_write_kb":17730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.d","version":"94'264","reported_seq":2543,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389126+0000","last_change":"2024-04-23T21:22:38.389428+0000","last_active":"2024-04-23T21:22:38.389126+0000","last_peered":"2024-04-23T21:22:38.389126+0000","last_clean":"2024-04-23T21:22:38.389126+0000","last_became_active":"2024-04-23T21:21:17.778797+0000","last_became_peered":"2024-04-23T21:21:17.778797+0000","last_unstale":"2024-04-23T21:22:38.389126+0000","last_undegraded":"2024-04-23T21:22:38.389126+0000","last_fullsized":"2024-04-23T21:22:38.389126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:54:15.435558+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0063591359999999996,"stat_sum":{"num_bytes":45966183,"num_objects":54,"num_object_clones":9,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":29,"num_read":2501,"num_read_kb":11452,"num_write":248,"num_write_kb":11068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.e","version":"94'506","reported_seq":1920,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.496079+0000","last_change":"2024-04-23T21:22:38.384963+0000","last_active":"2024-04-23T21:22:38.496079+0000","last_peered":"2024-04-23T21:22:38.496079+0000","last_clean":"2024-04-23T21:22:38.496079+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:38.496079+0000","last_undegraded":"2024-04-23T21:22:38.496079+0000","last_fullsized":"2024-04-23T21:22:38.496079+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":506,"log_dups_size":0,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:09:46.962009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320499999999999,"stat_sum":{"num_bytes":48671382,"num_objects":55,"num_object_clones":10,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":27,"num_read":3448,"num_read_kb":11204,"num_write":630,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.f","version":"94'896","reported_seq":9954,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.395078+0000","last_change":"2024-04-23T21:22:38.395242+0000","last_active":"2024-04-23T21:22:38.395078+0000","last_peered":"2024-04-23T21:22:38.395078+0000","last_clean":"2024-04-23T21:22:38.395078+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:38.395078+0000","last_undegraded":"2024-04-23T21:22:38.395078+0000","last_fullsized":"2024-04-23T21:22:38.395078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":896,"log_dups_size":0,"ondisk_log_size":896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:41:01.685567+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012161335000000001,"stat_sum":{"num_bytes":127130709,"num_objects":81,"num_object_clones":18,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":32,"num_read":13152,"num_read_kb":58727,"num_write":1272,"num_write_kb":16409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.10","version":"93'414","reported_seq":3099,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.483088+0000","last_change":"2024-04-23T21:22:38.438886+0000","last_active":"2024-04-23T21:22:38.483088+0000","last_peered":"2024-04-23T21:22:38.483088+0000","last_clean":"2024-04-23T21:22:38.483088+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:38.483088+0000","last_undegraded":"2024-04-23T21:22:38.483088+0000","last_fullsized":"2024-04-23T21:22:38.483088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:47:55.635384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047995299999999998,"stat_sum":{"num_bytes":101761897,"num_objects":74,"num_object_clones":17,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":30,"num_read":4138,"num_read_kb":33523,"num_write":481,"num_write_kb":20694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.11","version":"94'253","reported_seq":2188,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.389732+0000","last_change":"2024-04-23T21:22:38.390017+0000","last_active":"2024-04-23T21:22:38.389732+0000","last_peered":"2024-04-23T21:22:38.389732+0000","last_clean":"2024-04-23T21:22:38.389732+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:38.389732+0000","last_undegraded":"2024-04-23T21:22:38.389732+0000","last_fullsized":"2024-04-23T21:22:38.389732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:53:16.375004+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0077344930000000003,"stat_sum":{"num_bytes":91411180,"num_objects":66,"num_object_clones":17,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":27,"num_read":1874,"num_read_kb":20591,"num_write":283,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.12","version":"94'789","reported_seq":2928,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.454378+0000","last_change":"2024-04-23T21:22:38.384028+0000","last_active":"2024-04-23T21:22:38.454378+0000","last_peered":"2024-04-23T21:22:38.454378+0000","last_clean":"2024-04-23T21:22:38.454378+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:38.454378+0000","last_undegraded":"2024-04-23T21:22:38.454378+0000","last_fullsized":"2024-04-23T21:22:38.454378+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:26:45.125365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014285649999999999,"stat_sum":{"num_bytes":80276131,"num_objects":72,"num_object_clones":12,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":27,"num_read":5353,"num_read_kb":39496,"num_write":1086,"num_write_kb":21634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]},{"pgid":"3.13","version":"93'816","reported_seq":2742,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-23T21:22:38.382845+0000","last_change":"2024-04-23T21:22:38.383310+0000","last_active":"2024-04-23T21:22:38.382845+0000","last_peered":"2024-04-23T21:22:38.382845+0000","last_clean":"2024-04-23T21:22:38.382845+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:38.382845+0000","last_undegraded":"2024-04-23T21:22:38.382845+0000","last_fullsized":"2024-04-23T21:22:38.382845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:11:00.823606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084378100000000004,"stat_sum":{"num_bytes":83105146,"num_objects":70,"num_object_clones":14,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":5188,"num_read_kb":39132,"num_write":1248,"num_write_kb":24108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2385400573,"num_objects":2175,"num_object_clones":497,"num_object_copies":4350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2175,"num_whiteouts":1004,"num_read":177040,"num_read_kb":1010729,"num_write":17863,"num_write_kb":597645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":92680192,"data_stored":927288416,"data_compressed":55633033,"data_compressed_allocated":61050880,"data_compressed_original":899446117,"omap_allocated":0,"internal_metadata":0},"log_size":14685,"ondisk_log_size":14685,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65288,"kb_used_data":16060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94306552,"statfs":{"total":96636764160,"available":96569909248,"internally_reserved":0,"allocated":16445440,"data_stored":144462534,"data_compressed":9363404,"data_compressed_allocated":10027008,"data_compressed_original":139355245,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42648,"kb_used_data":7300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94329192,"statfs":{"total":96636764160,"available":96593092608,"internally_reserved":0,"allocated":7475200,"data_stored":64982199,"data_compressed":3626218,"data_compressed_allocated":4145152,"data_compressed_original":62472853,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49644,"kb_used_data":10136,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322196,"statfs":{"total":96636764160,"available":96585928704,"internally_reserved":0,"allocated":10379264,"data_stored":87504141,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83980,"kb_used_data":26436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94287860,"statfs":{"total":96636764160,"available":96550768640,"internally_reserved":0,"allocated":27070464,"data_stored":267140093,"data_compressed":15733233,"data_compressed_allocated":17158144,"data_compressed_original":258789727,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39128,"kb_used_data":7936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332712,"statfs":{"total":96636764160,"available":96596697088,"internally_reserved":0,"allocated":8126464,"data_stored":65599087,"data_compressed":3837804,"data_compressed_allocated":4386816,"data_compressed_original":62678344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528331,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46560,"kb_used_data":9876,"kb_used_omap":6,"kb_used_meta":36665,"kb_avail":94325280,"statfs":{"total":96636764160,"available":96589086720,"internally_reserved":0,"allocated":10113024,"data_stored":97825612,"data_compressed":6089726,"data_compressed_allocated":6176768,"data_compressed_original":94863024,"omap_allocated":6748,"internal_metadata":37545380},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69020,"kb_used_data":19792,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94302820,"statfs":{"total":96636764160,"available":96566087680,"internally_reserved":0,"allocated":20267008,"data_stored":199150039,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19156992,"data_stored":198632385,"data_compressed":11744042,"data_compressed_allocated":13111296,"data_compressed_original":193295362,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9138176,"data_stored":96862759,"data_compressed":6055668,"data_compressed_allocated":6131712,"data_compressed_original":94273200,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":64762460,"data_compressed":3818707,"data_compressed_allocated":4358144,"data_compressed_original":62364230,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":6676463,"data_compressed":354993,"data_compressed_allocated":483328,"data_compressed_original":6245965,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25718784,"data_stored":266049296,"data_compressed":15712561,"data_compressed_allocated":17125376,"data_compressed_original":258392415,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9117696,"data_stored":86851690,"data_compressed":5008581,"data_compressed_allocated":5742592,"data_compressed_original":83926209,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6336512,"data_stored":64152842,"data_compressed":3607499,"data_compressed_allocated":4116480,"data_compressed_original":62166931,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15212544,"data_stored":143300521,"data_compressed":9330982,"data_compressed_allocated":9981952,"data_compressed_original":138781805,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:42.896 INFO:tasks.thrashosds.thrasher:Setting 3.11 to [0, 4, 2, 6] 2024-04-23T21:22:42.896 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.11', '0', '4', '2', '6'] 2024-04-23T21:22:42.896 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.11 0 4 2 6 2024-04-23T21:22:42.913 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:42.914+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:42.965 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:42.965 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009320363, 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.966 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.970 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009282243000001, 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.971 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:42.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:42.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.975 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009259843999999, 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:42.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.978 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009240448, 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:42.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:42.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009220849999998, 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.984 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009200636999999, 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:42.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009168471000002, 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.992 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:42.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:42.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:42.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009143858000002, 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:42.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:42.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009122859999998, 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009102560999999, 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:43.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:43.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.009024771999997, 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.009 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:43.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.007529398000003, 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:43.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.007479508000003, 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.017 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:43.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:43.021 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:43.022 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.007455739000001, 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.023 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:43.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:43.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.007434732999997, 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:43.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:43.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:43.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:43.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.007415127999998, 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:43.042 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.043 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:43.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.044 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:43.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:43.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:43.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.052 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.991412979000003, 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:43.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:43.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.991394468000003, 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.060 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.991374022999999, 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:43.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:43.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:43.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:43.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 51.012296810999999, 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:43.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:43.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:43.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.070 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:43.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:43.072 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:43.072+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:43.172 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:43.173+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:43.216 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:43.216 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:43.216 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:43.216 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:43.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:43.273 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:43.274+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:43.367 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:43.367 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:43.367 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:43.367 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:43.367 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:43.374 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:43.375+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:43.375 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:43.413 INFO:teuthology.orchestra.run.smithi111.stderr:set 3.11 pg_upmap_items mapping to [0->4,2->6] 2024-04-23T21:22:43.425 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:22:43.475 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:43.476+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.616451960999999, 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.493 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:43.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:43.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.550033452000001, 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.498 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:43.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:43.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.550016071000002, 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.503 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:43.504 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:43.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.549998513999999, 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.507 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.548338318999999, 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.548316422999999, 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.548289072999999, 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:43.523 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.548264984999999, 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.548233926000002, 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.528 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:43.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.531 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.547033857999999, 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:43.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.547011425000001, 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.536 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.546987837, 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.540 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.546967897999998, 2024-04-23T21:22:43.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:43.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.548 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.546939423000001, 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:43.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:43.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.545556679000001, 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.540460055, 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:43.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:43.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:43.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.540113429000002, 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.528598525, 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.565 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:43.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.528578585999998, 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.569 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:43.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:43.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:43.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 23.527278315, 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:43.573 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:43.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:43.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:43.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:43.577 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:43.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:43.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:43.580 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:43.579+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:43.679 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:43.681+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:43.698 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:43.698 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:43.715 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:43.715 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:43.722 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:43.722 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:43.722 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:43.722 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:43.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:43.732 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:43.732 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:43.749 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:43.749 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:43.767 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:43.767 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:43.780 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:43.781+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:43.784 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:43.784 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:43.801 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:43.802 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:43.819 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:43.819 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:43.834 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:43.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:43.887 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:43.888+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:43.946 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:43.946 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:43.954 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:43.982 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:43.983+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:44.083 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:44.084+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:44.097 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.097 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.107 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:44.184 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:44.185+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:44.250 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.250 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.259 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:44.303 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:44.304+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:44.361 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.361 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.370 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:44.411 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:44.412+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:44.425 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:22:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.484 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.487 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:44.488+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:44.494 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:44.587 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:44.588+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:44.637 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.637 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.646 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:44.690 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:44.691+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:44.747 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.747 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.756 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:44.789 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:44.790+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:44.890 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:44.891+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:44.900 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:44.901 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:44.910 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:44.990 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:44.991+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:45.052 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.053 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.061 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:45.091 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:45.092+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:45.164 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.164 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.173 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:45.216 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:45.217+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:45.276 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.276 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.286 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:45.327 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:45.328+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:45.394 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:45.394+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:45.430 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.430 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.441 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:45.495 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:45.496+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:45.551 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.551 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.561 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:45.602 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:45.603+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:45.697 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:45.698+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:45.704 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.704 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.713 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:45.798 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:45.798+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:45.830 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:45.858 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:45.858 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:45.868 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:45.898 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:45.899+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:45.953 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:45.953 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:45.971 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:45.971 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:45.989 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:45.989 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:45.999 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:46.000+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:46.006 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:46.006 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:46.017 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.377625127999998, 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:46.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:46.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:46.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.021 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.377593021000003, 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.025 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.377569635, 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.026 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:46.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:46.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:46.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.377549328000001, 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.030 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.377529516000003, 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.034 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:46.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:46.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:46.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376458118000002, 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.038 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:46.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:46.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376424772999997, 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:46.042 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.046 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376393243999999, 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376365855000003, 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:46.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:46.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.055 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376353948999999, 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:46.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376335705000002, 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.060 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:46.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.064 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376325887999997, 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.065 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376315640999998, 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:46.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:46.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376305989999999, 2024-04-23T21:22:46.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.077 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.376272491000002, 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.078 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:46.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:46.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.372647067000003, 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:46.082 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.086 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.372629197999998, 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.087 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:46.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.372610469000001, 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:46.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.372592040999997, 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:46.096 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:46.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.372574135999997, 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:46.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:46.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:46.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:46.105 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:46.106 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:46.107 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:46.108 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:46.108 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:46.108 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:46.109 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:46.109 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:46.109 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:46.110 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:46.110 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:46.107+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:46.207 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:46.208+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:46.215 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.215 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:46.215 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:46.215 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:46.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:46.308 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:46.309+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:46.367 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.367 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:46.367 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:46.367 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:46.367 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:46.376 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:46.409 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:46.411+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:46.442 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:22:46.510 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:46.511+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.588072453999999, 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.545 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:46.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:46.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.588034334, 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.549 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:46.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:46.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.588011934999997, 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:46.553 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587992538999998, 2024-04-23T21:22:46.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:46.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.561 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587972940999997, 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:46.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587952727999998, 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587920562000001, 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587895949, 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587874951000003, 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:46.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.581 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587854651999997, 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.585 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.587776863000002, 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:46.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:46.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:46.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:46.589 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.586281489000001, 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.590 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:46.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:46.593 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.586231599000001, 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.594 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:46.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.586207829999999, 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:46.598 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.601 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:46.602 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.586186824000002, 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:46.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.606 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.586167219000004, 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.607 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:46.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:46.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:46.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.570165070000002, 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.611 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.570146559000001, 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.570126113999997, 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 54.591048901999997, 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.623 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.628 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.629 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:46.629 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:46.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:22:46.631 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:46.631+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:46.731 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:46.733+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:46.732 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:46.732 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:46.732 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:46.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-04-23T21:22:46.832 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:46.833+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:46.842 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.842 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:46.843 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:46.843 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:46.843 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:46.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-04-23T21:22:46.935 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:46.936+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:46.995 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:46.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:96433 3.5 3:a6915044:::rbd_data.1118d84f6bc8.000000000000005b:head [stat,delete] snapc 2b=[2b] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.119124918000001, 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054457868999999999, 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 96433 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875965+0000", 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5900000000000002e-07 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875968+0000", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3760000000000002e-06 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875975+0000", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.0620000000000003e-06 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.875980+0000", 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0629999999999996e-06 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925351+0000", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.049370879999999999 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.925479+0000", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012858499999999999 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.930423+0000", 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.004943444 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:46.999 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97536 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1568768~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.052706408999999, 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054494344, 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97536 2024-04-23T21:22:47.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942383+0000", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942384+0000", 2024-04-23T21:22:47.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3000000000000002e-07 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942386+0000", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.779e-06 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942391+0000", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4859999999999997e-06 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942395+0000", 2024-04-23T21:22:47.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2649999999999998e-06 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996703+0000", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054307719999999997 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996786+0000", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2816000000000001e-05 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996878+0000", 2024-04-23T21:22:47.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1947999999999994e-05 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97537 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1576960~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.052689028, 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054687979999999997, 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.004 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97537 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942401+0000", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942403+0000", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9130000000000001e-06 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942409+0000", 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3970000000000002e-06 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942413+0000", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1819999999999999e-06 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996905+0000", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054492152000000002 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.996997+0000", 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2025000000000001e-05 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997089+0000", 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1942999999999997e-05 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97538 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 1634304~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.052671471, 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054873298000000001, 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97538 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.942418+0000", 2024-04-23T21:22:47.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943932+0000", 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0015137460000000001 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943937+0000", 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6070000000000001e-06 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943945+0000", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5029999999999997e-06 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.943952+0000", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6370000000000002e-06 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997126+0000", 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053173858999999997 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997203+0000", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6909999999999994e-05 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.997292+0000", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9036000000000006e-05 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97544 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2002944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:47.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.051011276000001, 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054373652000000001, 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97544 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944079+0000", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0999999999999999e-07 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944081+0000", 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1009999999999999e-06 2024-04-23T21:22:47.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944089+0000", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1440000000000003e-06 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944094+0000", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9069999999999998e-06 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998292+0000", 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054198011999999997 2024-04-23T21:22:47.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998367+0000", 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5091000000000003e-05 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998452+0000", 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4987000000000005e-05 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97545 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2060288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.016 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.050989380000001, 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054538591999999997, 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97545 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944100+0000", 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944101+0000", 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3099999999999998e-07 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944103+0000", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0590000000000001e-06 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944109+0000", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9009999999999999e-06 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944115+0000", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998480+0000", 2024-04-23T21:22:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054364687000000002 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998554+0000", 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4239000000000005e-05 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998639+0000", 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5178000000000003e-05 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97546 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2125824~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.050962030000001, 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054699298, 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97546 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944128+0000", 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3300000000000003e-07 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944130+0000", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.119e-06 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944137+0000", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1380000000000004e-06 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944144+0000", 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.401e-06 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998667+0000", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054522582999999999 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998741+0000", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4293000000000002e-05 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998827+0000", 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6330999999999997e-05 2024-04-23T21:22:47.024 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97547 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2183168~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.050937942000001, 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054886936999999997, 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97547 2024-04-23T21:22:47.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944152+0000", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1699999999999999e-07 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944154+0000", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944161+0000", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4629999999999998e-06 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944166+0000", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1900000000000003e-06 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998854+0000", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054688417000000003 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.998936+0000", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1846000000000002e-05 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999039+0000", 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010257 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97548 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2248704~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.050906883, 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055120679999999998, 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97548 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.944183+0000", 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945304+0000", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.001121454 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945309+0000", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7589999999999997e-06 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945322+0000", 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2717999999999999e-05 2024-04-23T21:22:47.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945330+0000", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6029999999999999e-06 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999079+0000", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053749453000000003 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999181+0000", 2024-04-23T21:22:47.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102519 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999304+0000", 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00012217400000000001 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97550 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2371584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.049706815, 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054336369000000002, 2024-04-23T21:22:47.033 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97550 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945383+0000", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945384+0000", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9699999999999996e-07 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945386+0000", 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5790000000000002e-06 2024-04-23T21:22:47.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945393+0000", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2200000000000003e-06 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945399+0000", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2939999999999996e-06 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999546+0000", 2024-04-23T21:22:47.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054147721000000003 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999628+0000", 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1371000000000004e-05 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999719+0000", 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1687000000000005e-05 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97561 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2428928~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.049684381999999, 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054506318999999998, 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97561 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945405+0000", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945406+0000", 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945408+0000", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1069999999999999e-06 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945417+0000", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2150000000000007e-06 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945422+0000", 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2700000000000004e-06 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999747+0000", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054324961999999997 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999826+0000", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8505999999999997e-05 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999912+0000", 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5838e-05 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97562 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2486272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.049660794000001, 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054689271999999997, 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97562 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.042 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945429+0000", 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8000000000000001e-07 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945432+0000", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.21e-06 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945438+0000", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3049999999999998e-06 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945443+0000", 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9300000000000002e-06 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.999945+0000", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054502287000000003 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000031+0000", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5655000000000006e-05 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000118+0000", 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7504999999999997e-05 2024-04-23T21:22:47.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97563 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.049640855, 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054861884, 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.046 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97563 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945449+0000", 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9900000000000001e-07 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945456+0000", 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.601e-06 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945465+0000", 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2490000000000007e-06 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945471+0000", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.383e-06 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000147+0000", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054676791000000002 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000223+0000", 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5389000000000006e-05 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000311+0000", 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.8072000000000007e-05 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97564 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2551808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.049612379999999, 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055028711000000001, 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97564 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.945477+0000", 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946769+0000", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0012915699999999999 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946773+0000", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8909999999999997e-06 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946781+0000", 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:47.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946787+0000", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.122e-06 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000339+0000", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053551863999999998 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000416+0000", 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7046999999999996e-05 2024-04-23T21:22:47.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.000506+0000", 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.9884999999999997e-05 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97568 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 2797568~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.048229635999999, 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054427391999999998, 2024-04-23T21:22:47.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97568 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.946860+0000", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947785+0000", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00092466399999999995 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947789+0000", 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8190000000000002e-06 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947800+0000", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1671000000000001e-05 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.947807+0000", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6900000000000003e-06 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001129+0000", 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053321448 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001206+0000", 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7478999999999995e-05 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.001288+0000", 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1620999999999997e-05 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97590 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 4083712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.058 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.043133011999998, 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054371308, 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97590 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:47.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.951957+0000", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952253+0000", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000295949 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952257+0000", 2024-04-23T21:22:47.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952266+0000", 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5580000000000001e-06 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952272+0000", 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5319999999999997e-06 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006137+0000", 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053865841999999997 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006221+0000", 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3541999999999994e-05 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006328+0000", 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107228 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.062 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97619 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 6926336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.042786386, 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054424892000000002, 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97619 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952303+0000", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952695+0000", 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00039156400000000001 2024-04-23T21:22:47.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952700+0000", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7870000000000001e-06 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952709+0000", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0089999999999996e-06 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.952716+0000", 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7320000000000001e-06 2024-04-23T21:22:47.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006570+0000", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053853934999999999 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006652+0000", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2757999999999996e-05 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.006728+0000", 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6106999999999995e-05 2024-04-23T21:22:47.066 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97864 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3858432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.031271482000001, 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054451681000000002, 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97864 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963818+0000", 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963819+0000", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2e-07 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963821+0000", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0889999999999998e-06 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963826+0000", 2024-04-23T21:22:47.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4979999999999997e-06 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963831+0000", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1590000000000003e-06 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018113+0000", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054281851999999998 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018188+0000", 2024-04-23T21:22:47.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5234000000000005e-05 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018270+0000", 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2329000000000004e-05 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97865 3.1d 3:b91b339f:::rbd_data.111821c03bfd.0000000000000017:2a [read 3915776~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.031251543, 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054642380999999997, 2024-04-23T21:22:47.071 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97865 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.963838+0000", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964934+0000", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0010953250000000001 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964939+0000", 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.938e-06 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964950+0000", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0944999999999999e-05 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.964957+0000", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9920000000000001e-06 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018300+0000", 2024-04-23T21:22:47.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.053343730999999998 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018379+0000", 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9078000000000004e-05 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.018481+0000", 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000101372 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:97895 3.1d 3:ba96fc87:::rbd_data.1118f9168b6.0000000000000003:29 [read 7421952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e75)", 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 27.029951272000002, 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054387176000000002, 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 97895 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965139+0000", 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1100000000000002e-07 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965141+0000", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.86e-06 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965147+0000", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8799999999999996e-06 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:19.965151+0000", 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3880000000000002e-06 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019392+0000", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054241105999999997 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019458+0000", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6142999999999993e-05 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:20.019526+0000", 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7488000000000002e-05 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.080 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.080 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.081 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:47.082 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:47.082+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:47.182 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:22:47.182 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:47.183+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:47.232 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.232 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:137145 3.1e 3:78980a10:::rbd_object_map.111821c03bfd:head [call lock.unlock in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:47.233045+0000", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00083147400000000002, 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00084736399999999997, 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "waiting for sub ops", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 137145 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233045+0000", 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233045+0000", 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233046+0000", 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233049+0000", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.6390000000000001e-06 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233056+0000", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7440000000000007e-06 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233060+0000", 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7780000000000001e-06 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233098+0000", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7484000000000002e-05 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233172+0000", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4702000000000004e-05 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 2", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233332+0000", 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000159584 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233775+0000", 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00044272499999999999 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_repop_reply(client.4376.0:137145 3.1e e103/88)", 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:47.233893+0000", 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 4294967295.9999838, 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9277999999999997e-05, 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:47.237 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233893+0000", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233893+0000", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233893+0000", 2024-04-23T21:22:47.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7399999999999999e-07 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233896+0000", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1339999999999998e-06 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233903+0000", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8420000000000005e-06 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233907+0000", 2024-04-23T21:22:47.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.608e-06 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:47.233947+0000", 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9700000000000003e-05 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 2 2024-04-23T21:22:47.240 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:47.286 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:47.287+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:47.350 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:47.350 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:47.351 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:47.351 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.359 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:47.400 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:47.401+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:47.484 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:47.485+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.505 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.851001185000001, 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850980061000001, 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.510 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:47.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850950793999999, 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.514 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:47.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:47.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:47.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850930032000001, 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:47.518 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:47.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.522 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.523 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850911005999997, 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:47.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:47.525 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.526 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.527 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850892166999998, 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.528 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.529 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.530 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.531 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850858764999998, 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.532 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.533 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:47.534 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:47.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850839018999999, 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.536 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:47.537 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.538 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.539 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850820916000004, 2024-04-23T21:22:47.540 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.541 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.542 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.543 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850801715000003, 2024-04-23T21:22:47.544 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.545 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:47.546 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:47.547 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.548 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850776316000001, 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.549 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.550 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:47.551 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.552 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850751191000001, 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.553 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.554 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.555 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.556 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850726180000002, 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.557 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.559 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.560 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850702611000003, 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:47.561 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:47.562 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:47.563 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:47.564 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850679456000002, 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.565 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:47.566 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:47.567 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:47.568 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850658713000001, 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.569 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:47.570 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:47.571 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:47.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850640411999997, 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:47.573 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:47.577 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850622831999999, 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:47.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:47.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:47.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.581 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.850593308999997, 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 47.111338244999999, 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.586 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:47.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:47.591 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:47.593 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:47.593+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:47.693 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:47.694+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:47.736 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:47.736 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:47.736 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:47.736 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:47.794 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:47.795+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:47.848 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:47.848 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:47.849 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:47.857 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:47.898 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:47.899+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:47.997 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:47.998+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:48.001 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 42.873947413000003, 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:48.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.007 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073973669000001, 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:48.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:22:48.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:22:48.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073951009999998, 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:22:48.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.07392449, 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.016 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073904425999999, 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.020 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.021 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:22:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073611755999998, 2024-04-23T21:22:48.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:22:48.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:22:48.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073601850999999, 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.031 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073299071000001, 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.033 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:22:48.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:22:48.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:22:48.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073262391, 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.037 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073240768000002, 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.041 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.042 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.043 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:48.045 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073220523, 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:48.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:22:48.049 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.073019206000001, 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.050 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:22:48.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:22:48.053 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.072584535000001, 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:22:48.054 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.055 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.056 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.057 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.058 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.07231771, 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.059 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:22:48.060 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.061 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.062 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.072295509, 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.063 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:48.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:22:48.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:22:48.066 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.072275919999999, 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.067 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.068 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:22:48.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.071 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.071937607999999, 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.075 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.071917026000001, 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:22:48.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.071898002000001, 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:22:48.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:22:48.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 29.071671838, 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:48.084 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:48.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:48.088 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:48.089 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:48.098 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:22:48.099+0000 7fbcb1fbf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 32384) UID: 0 2024-04-23T21:22:48.111 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:48.199 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:48.200+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:48.203 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:48.203 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:48.203 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:48.203 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:48.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:48.265 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:48.265 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:48.286 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:48.287 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:48.301 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:48.302+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:48.304 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:48.304 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:48.323 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:48.323 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:48.326 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.327 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.336 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:48.341 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:48.341 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:48.358 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5: {} 2024-04-23T21:22:48.359 INFO:teuthology.orchestra.run.smithi111.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-23T21:22:48.376 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:48.377 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:48.394 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:48.394 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:48.401 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:48.403+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:48.432 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:48.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:48.432 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 4, 5, 6, 7, 1] 2024-04-23T21:22:48.433 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-23T21:22:48.433 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:22:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.447 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.457 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:48.462 INFO:tasks.ceph.osd.5:Stopped 2024-04-23T21:22:48.462 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-04-23T21:22:48.462 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:22:48.463 DEBUG:teuthology.orchestra.run.smithi192:> type ceph-objectstore-tool 2024-04-23T21:22:48.503 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:48.504+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:48.517 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-04-23T21:22:48.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-04-23T21:22:48.558 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.558 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:48.610 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:48.611+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:48.669 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.670 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.678 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:48.720 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:48.721+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.d 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.c 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.2 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.5 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.1d 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.1f 2024-04-23T21:22:48.756 INFO:teuthology.orchestra.run.smithi192.stdout:3.1c 2024-04-23T21:22:48.757 INFO:teuthology.orchestra.run.smithi192.stdout:3.17 2024-04-23T21:22:48.757 INFO:teuthology.orchestra.run.smithi192.stdout:3.16 2024-04-23T21:22:48.757 INFO:teuthology.orchestra.run.smithi192.stdout:1.4 2024-04-23T21:22:48.757 INFO:teuthology.orchestra.run.smithi192.stdout:1.1 2024-04-23T21:22:48.806 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:48.807+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:48.821 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.822 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.830 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:48.907 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:48.908+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:48.974 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:48.974 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:48.983 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:49.023 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 3.d --file /var/log/ceph/exp.3.d.5 2024-04-23T21:22:49.027 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:49.028+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:49.085 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.086 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.094 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:49.136 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:49.137+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:49.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:22:49.245 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:49.246+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:49.246 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.246 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.256 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:49.284 INFO:teuthology.orchestra.run.smithi192.stderr:Exporting 3.d info 3.d( v 103'270 (0'0,103'270] local-lis/les=18/19 n=48 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-23T21:22:49.287 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b0045f95:::rbd_data.11187efdbcb.00000000000000b8:head# 2024-04-23T21:22:49.287 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b02dfd37:::rbd_data.1118d84f6bc8.0000000000000061:head# 2024-04-23T21:22:49.287 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b10419f7:::rbd_data.111841bbf451.0000000000000133:head# 2024-04-23T21:22:49.287 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b11c9a16:::rbd_data.1118139b0f86.000000000000004b:head# 2024-04-23T21:22:49.289 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b142947d:::rbd_data.11187efdbcb.00000000000000f9:head# 2024-04-23T21:22:49.289 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b18cc5ef:::rbd_data.111832c889c4.0000000000000009:head# 2024-04-23T21:22:49.295 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b1bab69f:::rbd_header.11189d22003d:head# 2024-04-23T21:22:49.295 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b1d4f0d6:::rbd_data.111841bbf451.000000000000009e:head# 2024-04-23T21:22:49.295 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b22dddc2:::rbd_data.111821c03bfd.000000000000002a:head# 2024-04-23T21:22:49.295 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b242757d:::rbd_data.11185f1e06d0.000000000000000d:14# 2024-04-23T21:22:49.295 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b242757d:::rbd_data.11185f1e06d0.000000000000000d:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b24f968f:::rbd_data.111841bbf451.0000000000000101:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b2644df1:::rbd_data.11187efdbcb.00000000000000a7:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b3022d48:::rbd_data.11187efdbcb.00000000000000fb:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b339b192:::rbd_data.111841bbf451.00000000000000da:26# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b339b192:::rbd_data.111841bbf451.00000000000000da:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b3421a8d:::rbd_data.111841bbf451.00000000000000b3:head# 2024-04-23T21:22:49.296 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b368b598:::rbd_data.1118732f032e.0000000000000008:head# 2024-04-23T21:22:49.297 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b36a456a:::rbd_data.11187efdbcb.00000000000000c6:head# 2024-04-23T21:22:49.297 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b372b423:::rbd_data.11183c92f5c.0000000000000004:5# 2024-04-23T21:22:49.297 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b372b423:::rbd_data.11183c92f5c.0000000000000004:head# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b3ba4ef8:::rbd_object_map.1118f7f89011.0000000000000006:head# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b43bade4:::rbd_id.image_client.0-clone35:head# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b45be3d8:::rbd_data.111841bbf451.00000000000000c6:head# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b4743de8:::rbd_data.11187efdbcb.00000000000000e5:head# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b492ad03:::rbd_data.1118dbdf22af.0000000000000018:a# 2024-04-23T21:22:49.299 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b492ad03:::rbd_data.1118dbdf22af.0000000000000018:head# 2024-04-23T21:22:49.300 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b4b358f5:::rbd_data.111825c5a0b7.000000000000000f:head# 2024-04-23T21:22:49.308 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b4ddd7fa:::rbd_object_map.1118c9f76dd8.0000000000000004:head# 2024-04-23T21:22:49.308 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b4e98a40:::rbd_data.111841bbf451.000000000000015a:head# 2024-04-23T21:22:49.308 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b5008dd3:::rbd_data.11187efdbcb.0000000000000071:1b# 2024-04-23T21:22:49.309 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b5008dd3:::rbd_data.11187efdbcb.0000000000000071:head# 2024-04-23T21:22:49.309 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b5449feb:::rbd_data.111841bbf451.0000000000000135:head# 2024-04-23T21:22:49.310 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b5f2e565:::rbd_data.1118e8093af9.0000000000000044:head# 2024-04-23T21:22:49.310 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b610f7b3:::rbd_data.111841bbf451.00000000000000cc:head# 2024-04-23T21:22:49.310 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6663b33:::rbd_data.1118732f032e.0000000000000006:head# 2024-04-23T21:22:49.310 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6b4ee1d:::rbd_data.11183e0a89c0.0000000000000005:16# 2024-04-23T21:22:49.312 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6b4ee1d:::rbd_data.11183e0a89c0.0000000000000005:head# 2024-04-23T21:22:49.312 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6ce9c41:::rbd_data.111841bbf451.000000000000017b:head# 2024-04-23T21:22:49.312 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6e86ae8:::rbd_data.11187efdbcb.0000000000000093:head# 2024-04-23T21:22:49.312 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6ecd2ce:::rbd_data.11183e0a89c0.0000000000000003:16# 2024-04-23T21:22:49.314 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b6ecd2ce:::rbd_data.11183e0a89c0.0000000000000003:head# 2024-04-23T21:22:49.316 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b701e044:::rbd_data.111828d511a8.0000000000000001:head# 2024-04-23T21:22:49.319 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b71d943d:::rbd_data.111841bbf451.00000000000000b5:head# 2024-04-23T21:22:49.319 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b72b3450:::rbd_data.1118d84f6bc8.000000000000002f:head# 2024-04-23T21:22:49.319 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b7528495:::rbd_data.11187efdbcb.0000000000000066:head# 2024-04-23T21:22:49.320 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b776f8e0:::rbd_data.111830ac55a1.000000000000000f:head# 2024-04-23T21:22:49.320 INFO:teuthology.orchestra.run.smithi192.stderr:Read #3:b7b3f847:::rbd_data.111841bbf451.00000000000000ed:head# 2024-04-23T21:22:49.320 INFO:teuthology.orchestra.run.smithi192.stderr:Export successful 2024-04-23T21:22:49.342 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:49.343+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:49.402 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.402 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:49.452 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:49.453+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:49.513 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:49.514+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:49.553 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.553 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.562 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:49.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 3.d 2024-04-23T21:22:49.614 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:49.615+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:49.665 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.665 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:49.718 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:49.719+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:49.778 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.778 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:49.816 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:49.817+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:49.918 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:49.919+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:49.930 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:49.931 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:49.939 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:50.019 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:50.020+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:50.043 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:50.044 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:50.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:50.136 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:50.137+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:50.195 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:22:50.215 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:50.215 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:50.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:50.254 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:50.255+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:50.321 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:50.322+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:50.370 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:50.370 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:50.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:50.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:50.451 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:50.452+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:50.483 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:50.483 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:50.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:50.524 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:50.525+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:50.526 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:50.527 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:50.544 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:50.544 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:50.560 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:50.560 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:50.577 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:50.577 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:50.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:50.594 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:50.595 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:22:50.595 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:50.595 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:50.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:50.612 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:50.612 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:50.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:50.629 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:22:50.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:22:50.647 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:50.648+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:50.706 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:50.707 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:50.707 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:50.707 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:50.707 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:50.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:50.726 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:50.727+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:50.827 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:50.828+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:50.865 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 45.737468259000003, 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:22:50.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.871 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937494515000001, 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:50.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:22:50.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:22:50.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937471855999998, 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:22:50.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937445336, 2024-04-23T21:22:50.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.886 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937425271999999, 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.887 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:50.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:22:50.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:22:50.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:22:50.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:22:50.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:22:50.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:22:50.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.896 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.896 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937132601999998, 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.897 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.898 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:22:50.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:22:50.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:22:50.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.905 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.906 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.937122697, 2024-04-23T21:22:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:22:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.907 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.908 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:22:50.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:22:50.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.915 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936819917000001, 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.916 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:22:50.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936783237, 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.921 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:22:50.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:22:50.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936761614000002, 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.929 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936741369, 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.934 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:22:50.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936540052000002, 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.939 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:22:50.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:22:50.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.936105381000001, 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:22:50.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:22:50.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935838556, 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:22:50.949 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:22:50.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:22:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:22:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:22:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:22:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.956 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935816355, 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:22:50.958 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.959 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:50.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:22:50.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:22:50.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.967 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935796765999999, 2024-04-23T21:22:50.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:22:50.969 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.969 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.969 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.970 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.970 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:22:50.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.970 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:50.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:22:50.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:22:50.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:22:50.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:22:50.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935458453999999, 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:22:50.982 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935437872000001, 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.983 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:22:50.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.988 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:50.989 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935418848000001, 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:50.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.992 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 31.935192684, 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:50.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:50.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:50.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:50.998 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:51.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:51.001 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:51.000+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:51.100 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:51.101+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:51.105 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.105 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:51.105 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:51.106 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.114 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:51.136 INFO:teuthology.orchestra.run.smithi192.stdout: marking collection for removal 2024-04-23T21:22:51.137 INFO:teuthology.orchestra.run.smithi192.stdout:setting '_remove' omap key 2024-04-23T21:22:51.137 INFO:teuthology.orchestra.run.smithi192.stdout:finish_remove_pgs 3.d_head removing 3.d 2024-04-23T21:22:51.140 INFO:teuthology.orchestra.run.smithi192.stdout:Remove successful 2024-04-23T21:22:51.202 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:51.202+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.222 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:51.222 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:51.222 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:51.222 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:51.302 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:51.304+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:51.343 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 62.402217806000003, 2024-04-23T21:22:51.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:22:51.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:22:51.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:22:51.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.350 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572818877000003, 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:51.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:22:51.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:22:51.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572795943000003, 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.355 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.356 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:22:51.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572770624, 2024-04-23T21:22:51.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:22:51.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:22:51.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:22:51.364 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.365 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572751904999997, 2024-04-23T21:22:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:22:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.366 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:22:51.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:22:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:22:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:22:51.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572709224, 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.376 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:22:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:22:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:22:51.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:22:51.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.383 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:22:51.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.385 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572683083999998, 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.386 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:22:51.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:22:51.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:22:51.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572665166, 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.396 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572633787000001, 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.397 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:51.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:22:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:22:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:22:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:22:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572616658000001, 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:22:51.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572588119999999, 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.408 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572545134999999, 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.414 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.415 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:22:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:22:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:22:51.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:22:51.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572517898000001, 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.422 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572494110999997, 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.426 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572468868999998, 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:22:51.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:22:51.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572450103000001, 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.435 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572393142999999, 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:22:51.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:22:51.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:22:51.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572371408999999, 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.444 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 37.572347188000002, 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.448 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:22:51.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:22:51.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 18.143078132999999, 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:22:51.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:22:51.459 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.460 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.460 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.460 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.461 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:51.462+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:51.462 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:51.562 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:51.563+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:51.606 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.607 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:51.607 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:51.607 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.615 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:51.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.d.5 2024-04-23T21:22:51.693 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:51.694+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:51.758 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.758 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:51.758 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:51.758 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:51.759 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.763 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:51.764+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:51.767 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:51.864 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:51.865+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441855175000001, 2024-04-23T21:22:51.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:51.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:51.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:51.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441832364000007, 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.877 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.880 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.881 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441812753999997, 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.882 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:51.886 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441793891000003, 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.887 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:51.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:51.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441774596000002, 2024-04-23T21:22:51.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:51.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441748035000003, 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.899 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441706804999995, 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.900 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441673077999994, 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.904 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:51.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:51.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441649669, 2024-04-23T21:22:51.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:51.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:51.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:51.913 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441616418999999, 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.914 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:51.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:51.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441593026000007, 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.919 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:51.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441572476999994, 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.924 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:51.929 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.441546716999994, 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:51.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:51.934 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.440708701000005, 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.935 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:51.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:51.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.440685369999997, 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:51.940 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:51.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.439798077000006, 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:51.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:51.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.439564497000006, 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.950 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:51.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.439447799000007, 2024-04-23T21:22:51.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:51.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.958 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:51.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.960 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.439073385, 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:51.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:51.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 82.437716499999993, 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:51.965 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:51.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:51.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:51.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:51.970 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:51.972 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:51.972 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:51.972+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:52.074 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:52.075+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:52.115 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:52.115 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:52.115 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:52.115 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:52.125 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:52.176 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:52.177+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:52.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:22:52.238 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:52.238 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:52.238 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:52.238 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:52.238 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:52.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:52.278 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:52.279+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:52.380 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:52.381+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:52.397 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:52.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166173141000002, 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.399 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.400 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:52.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:52.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:52.407 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.408 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166143473999995, 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:52.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.417 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:52.418 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166124065999995, 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.419 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:52.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:52.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:52.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166102257999995, 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:52.427 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:52.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:52.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:52.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166082654999997, 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.434 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:52.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:52.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:52.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.443 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166063050000005, 2024-04-23T21:22:52.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:52.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:52.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:52.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166042208999997, 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:52.455 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:52.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:52.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.461 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.166012944000002, 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:52.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:52.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:52.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.465 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 66.165981751000004, 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.357898739999996, 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:52.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:52.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354670522000006, 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354648140999998, 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.482 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.486 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354628863000002, 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.490 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354611429000002, 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:52.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:52.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:52.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:52.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354593979000001, 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.495 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354576604000002, 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:52.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354558561999994, 2024-04-23T21:22:52.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.507 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.354533441000001, 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.508 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:52.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:52.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:52.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.352413869000003, 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:52.513 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.514 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.517 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 52.012790877999997, 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.521 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:52.522 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:52.522 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:52.523 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:52.524 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:52.524+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:52.623 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:52.624+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:52.633 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:52.670 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:52.670 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:52.670 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:52.670 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:52.679 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:52.755 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:52.756 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:52.756 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:52.756+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:52.773 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:52.773 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:52.790 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:52.790 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:52.795 INFO:teuthology.orchestra.run.smithi192.stdout:get_pg_num_history pg_num_history pg_num_history(e104 pg_nums {1={14=8},2={15=1},3={18=32}} deleted_pools ) 2024-04-23T21:22:52.795 INFO:teuthology.orchestra.run.smithi192.stdout:Importing pgid 3.d 2024-04-23T21:22:52.801 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b0045f95:::rbd_data.11187efdbcb.00000000000000b8:head# 2024-04-23T21:22:52.801 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.802 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b02dfd37:::rbd_data.1118d84f6bc8.0000000000000061:head# 2024-04-23T21:22:52.802 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 2b=[]:{} 2024-04-23T21:22:52.802 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b10419f7:::rbd_data.111841bbf451.0000000000000133:head# 2024-04-23T21:22:52.802 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.803 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b11c9a16:::rbd_data.1118139b0f86.000000000000004b:head# 2024-04-23T21:22:52.804 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.805 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b142947d:::rbd_data.11187efdbcb.00000000000000f9:head# 2024-04-23T21:22:52.805 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.806 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b18cc5ef:::rbd_data.111832c889c4.0000000000000009:head# 2024-04-23T21:22:52.806 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:52.807 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:52.810 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.815 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b1bab69f:::rbd_header.11189d22003d:head# 2024-04-23T21:22:52.815 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.815 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b1d4f0d6:::rbd_data.111841bbf451.000000000000009e:head# 2024-04-23T21:22:52.815 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.816 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b22dddc2:::rbd_data.111821c03bfd.000000000000002a:head# 2024-04-23T21:22:52.816 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 2a=[]:{} 2024-04-23T21:22:52.816 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b242757d:::rbd_data.11185f1e06d0.000000000000000d:14# 2024-04-23T21:22:52.817 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b242757d:::rbd_data.11185f1e06d0.000000000000000d:head# 2024-04-23T21:22:52.817 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 14=[]:{14=[14]} 2024-04-23T21:22:52.817 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b24f968f:::rbd_data.111841bbf451.0000000000000101:head# 2024-04-23T21:22:52.817 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.818 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b2644df1:::rbd_data.11187efdbcb.00000000000000a7:head# 2024-04-23T21:22:52.818 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.818 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b3022d48:::rbd_data.11187efdbcb.00000000000000fb:head# 2024-04-23T21:22:52.818 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.819 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b339b192:::rbd_data.111841bbf451.00000000000000da:26# 2024-04-23T21:22:52.819 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b339b192:::rbd_data.111841bbf451.00000000000000da:head# 2024-04-23T21:22:52.819 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{26=[26]} 2024-04-23T21:22:52.820 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b3421a8d:::rbd_data.111841bbf451.00000000000000b3:head# 2024-04-23T21:22:52.820 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.821 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b368b598:::rbd_data.1118732f032e.0000000000000008:head# 2024-04-23T21:22:52.821 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.821 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b36a456a:::rbd_data.11187efdbcb.00000000000000c6:head# 2024-04-23T21:22:52.821 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.822 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b372b423:::rbd_data.11183c92f5c.0000000000000004:5# 2024-04-23T21:22:52.824 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:52.825+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:52.825 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b372b423:::rbd_data.11183c92f5c.0000000000000004:head# 2024-04-23T21:22:52.825 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 5=[]:{5=[5]} 2024-04-23T21:22:52.825 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:52.825 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:52.826 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:22:52.826 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:52.826 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:52.827 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b3ba4ef8:::rbd_object_map.1118f7f89011.0000000000000006:head# 2024-04-23T21:22:52.827 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.830 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b43bade4:::rbd_id.image_client.0-clone35:head# 2024-04-23T21:22:52.830 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.832 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b45be3d8:::rbd_data.111841bbf451.00000000000000c6:head# 2024-04-23T21:22:52.833 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.833 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b4743de8:::rbd_data.11187efdbcb.00000000000000e5:head# 2024-04-23T21:22:52.833 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:52.836 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b492ad03:::rbd_data.1118dbdf22af.0000000000000018:a# 2024-04-23T21:22:52.836 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b492ad03:::rbd_data.1118dbdf22af.0000000000000018:head# 2024-04-23T21:22:52.837 INFO:teuthology.orchestra.run.smithi192.stdout:snapset a=[]:{a=[a]} 2024-04-23T21:22:52.839 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b4b358f5:::rbd_data.111825c5a0b7.000000000000000f:head# 2024-04-23T21:22:52.843 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:52.844 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:52.845 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.853 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b4ddd7fa:::rbd_object_map.1118c9f76dd8.0000000000000004:head# 2024-04-23T21:22:52.853 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.855 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b4e98a40:::rbd_data.111841bbf451.000000000000015a:head# 2024-04-23T21:22:52.855 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.855 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b5008dd3:::rbd_data.11187efdbcb.0000000000000071:1b# 2024-04-23T21:22:52.858 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b5008dd3:::rbd_data.11187efdbcb.0000000000000071:head# 2024-04-23T21:22:52.858 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{1b=[1b]} 2024-04-23T21:22:52.859 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b5449feb:::rbd_data.111841bbf451.0000000000000135:head# 2024-04-23T21:22:52.859 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.861 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:52.861 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:52.862 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b5f2e565:::rbd_data.1118e8093af9.0000000000000044:head# 2024-04-23T21:22:52.862 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 32=[]:{} 2024-04-23T21:22:52.864 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b610f7b3:::rbd_data.111841bbf451.00000000000000cc:head# 2024-04-23T21:22:52.864 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:22:52.865 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:22:52.867 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6663b33:::rbd_data.1118732f032e.0000000000000006:head# 2024-04-23T21:22:52.867 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.867 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6b4ee1d:::rbd_data.11183e0a89c0.0000000000000005:16# 2024-04-23T21:22:52.869 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6b4ee1d:::rbd_data.11183e0a89c0.0000000000000005:head# 2024-04-23T21:22:52.869 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 16=[]:{16=[16]} 2024-04-23T21:22:52.870 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6ce9c41:::rbd_data.111841bbf451.000000000000017b:head# 2024-04-23T21:22:52.870 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.870 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6e86ae8:::rbd_data.11187efdbcb.0000000000000093:head# 2024-04-23T21:22:52.870 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.871 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6ecd2ce:::rbd_data.11183e0a89c0.0000000000000003:16# 2024-04-23T21:22:52.873 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b6ecd2ce:::rbd_data.11183e0a89c0.0000000000000003:head# 2024-04-23T21:22:52.874 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 16=[]:{16=[16]} 2024-04-23T21:22:52.875 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b701e044:::rbd_data.111828d511a8.0000000000000001:head# 2024-04-23T21:22:52.876 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.879 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b71d943d:::rbd_data.111841bbf451.00000000000000b5:head# 2024-04-23T21:22:52.879 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.879 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b72b3450:::rbd_data.1118d84f6bc8.000000000000002f:head# 2024-04-23T21:22:52.879 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 2b=[]:{} 2024-04-23T21:22:52.880 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b7528495:::rbd_data.11187efdbcb.0000000000000066:head# 2024-04-23T21:22:52.880 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 1b=[]:{} 2024-04-23T21:22:52.881 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b776f8e0:::rbd_data.111830ac55a1.000000000000000f:head# 2024-04-23T21:22:52.882 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 0=[]:{} 2024-04-23T21:22:52.882 INFO:teuthology.orchestra.run.smithi192.stdout:Write #3:b7b3f847:::rbd_data.111841bbf451.00000000000000ed:head# 2024-04-23T21:22:52.882 INFO:teuthology.orchestra.run.smithi192.stdout:snapset 26=[]:{} 2024-04-23T21:22:52.882 INFO:teuthology.orchestra.run.smithi192.stdout:write_pg epoch 104 info 3.d( v 103'270 (0'0,103'270] local-lis/les=18/19 n=48 ec=18/18 lis/c=18/18 les/c/f=19/19/0 sis=18) 2024-04-23T21:22:52.883 INFO:teuthology.orchestra.run.smithi192.stdout:Import successful 2024-04-23T21:22:52.926 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:52.926+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:52.946 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:52.946 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:52.955 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:53.027 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:53.028+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:53.065 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.065 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.073 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:53.128 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:53.129+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:53.175 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.176 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:53.206 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:22:53.229 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:53.230+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:53.298 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.298 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.310 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:53.350 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:53.351+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:53.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /var/log/ceph/exp.3.d.5 2024-04-23T21:22:53.430 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:53.431+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:53.456 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.456 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.465 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:53.550 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:53.551+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:53.608 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.609 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:53.631 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:53.632+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:53.732 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:53.733+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:53.761 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.761 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:53.854 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:53.855+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:53.914 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:53.914 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:53.923 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:53.964 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:53.965+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:54.035 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:54.036+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:54.065 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.065 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.074 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:54.159 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:54.160+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:54.218 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.219 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:54.271 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:54.272+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:54.338 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:54.338+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:54.351 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.352 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.361 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:54.438 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:54.439+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:54.464 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.464 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:54.557 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:54.558+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:54.615 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.615 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.624 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:54.665 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:54.666+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:54.741 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:54.743+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:54.767 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.767 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:54.860 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:54.861+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:54.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:54.921 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:54.921 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:54.931 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:54.974 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:54.975+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:54.988 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:54.988 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:55.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:55.006 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:55.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:55.023 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:55.037 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607341032999997, 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.038 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607318222000004, 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.042 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.043 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:22:55.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:22:55.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607298611999994, 2024-04-23T21:22:55.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:55.050 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607279749, 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:55.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:22:55.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:22:55.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607260453999999, 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:22:55.058 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607233893, 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.059 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607192663000006, 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.063 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:22:55.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607158936000005, 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:22:55.067 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:55.071 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607135526999997, 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.075 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607102276999996, 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:55.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.079 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607078884000003, 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.080 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:22:55.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:22:55.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:22:55.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607058335000005, 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.084 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.085 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.607032575000005, 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:22:55.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:22:55.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.606194559000002, 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:22:55.092 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.096 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.606171227999994, 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.605283935000003, 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:22:55.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:22:55.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:22:55.104 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.605050355000003, 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:22:55.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.604933657000004, 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.109 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.604559242999997, 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 85.603202358000004, 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.117 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:22:55.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:22:55.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:22:55.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.121 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.122 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:55.122 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:55.123 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:55.124 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:55.124 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:55.124 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:22:55.124 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:55.125 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:55.125 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:22:55.126 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:55.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:22:55.126 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:55.124+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:55.223 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:22:55.224 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:55.225+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:55.273 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.273 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:55.273 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:55.273 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:55.326 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:55.328+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:55.394 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.394 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:55.394 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:55.394 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:55.394 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:55.445 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:55.446+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:55.526 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:55.528+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.318109390000004, 2024-04-23T21:22:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:22:55.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:22:55.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.318079722999997, 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.554 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:55.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.558 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.318060314999997, 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.559 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:22:55.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:22:55.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:22:55.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:22:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.318038506999997, 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.568 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:55.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:55.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:22:55.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:55.575 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.318018903999999, 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.576 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:55.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:22:55.581 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.317999298999993, 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:22:55.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.317978457999999, 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.591 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.317949193000004, 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:55.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:22:55.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.595 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.317918000000006, 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.596 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:22:55.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.509834988999998, 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.600 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.603 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:55.604 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506606770999994, 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:22:55.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:22:55.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.50658439, 2024-04-23T21:22:55.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:22:55.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:22:55.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506565112000004, 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.614 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.618 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506547678000004, 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:55.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:22:55.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:22:55.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.623 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506530228000003, 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:22:55.624 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.625 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506512853000004, 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.628 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:22:55.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506494810999996, 2024-04-23T21:22:55.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:22:55.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.506469690000003, 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.637 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.640 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 67.504350118000005, 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.641 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:22:55.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.645 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 55.164727126999999, 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:55.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:22:55.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:22:55.650 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.651 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:55.651 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:55.651 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.652 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:55.653 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:55.653+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:55.754 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:55.755+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:55.796 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.796 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:55.796 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:22:55.796 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.805 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:55.889 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:55.890+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:55.948 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:55.948 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:22:55.948 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:22:55.948 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:22:55.948 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:55.955 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:55.956+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:55.956 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:56.057 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:56.058+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:56.100 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:22:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:22:56.100 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.460509521000006, 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:22:56.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:22:56.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:22:56.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.460477413999996, 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.105 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.460454028000001, 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.109 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:22:56.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:22:56.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.460433721000001, 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.113 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:22:56.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.460413908999996, 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.117 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:56.121 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459342511000003, 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:56.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:22:56.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.125 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459309165999997, 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459277637, 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:22:56.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459250248000004, 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.134 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:56.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.136 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459238342000006, 2024-04-23T21:22:56.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:22:56.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:22:56.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459220098000003, 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.146 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459210280999997, 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.150 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459200034000006, 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:22:56.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:22:56.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459190383000006, 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.155 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.156 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:22:56.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.459156883999995, 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.160 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:22:56.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:22:56.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:22:56.163 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.455531460000003, 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.164 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.455513590999999, 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.168 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:56.172 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.455494861999995, 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:22:56.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:22:56.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.176 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.455476434000005, 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:56.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.180 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 69.455458528999998, 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.181 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:22:56.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:22:56.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:22:56.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:56.185 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:56.187 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:56.187 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:56.188+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:56.234 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:22:56.287 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:56.288+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:56.311 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:56.311 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:56.311 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:56.311 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:56.321 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:56.387 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:56.388+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:56.431 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:56.431 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:56.431 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:56.431 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:56.431 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:56.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:56.488 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:56.489+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630344581000003, 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.587 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:56.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:22:56.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630306461000004, 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.592 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:22:56.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.596 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630284062000001, 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630264666000002, 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:22:56.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:22:56.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630245067999994, 2024-04-23T21:22:56.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:22:56.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:22:56.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630224854999994, 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.611 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:22:56.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.615 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630192688999998, 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:22:56.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:22:56.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:22:56.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:22:56.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630168076000004, 2024-04-23T21:22:56.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.624 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630147077999993, 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:22:56.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630126778999994, 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.630 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.634 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.630048990000006, 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:22:56.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:22:56.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:22:56.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:22:56.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:22:56.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:22:56.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.628553616000005, 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:22:56.644 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:22:56.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.651 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.628503726000005, 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:22:56.652 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.653 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:22:56.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:22:56.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.628479956999996, 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:22:56.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.628458950999999, 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:22:56.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.628439345999993, 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:56.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.612437197000006, 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.612418685999998, 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:22:56.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:22:56.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:56.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.612398240999994, 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:56.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 64.633321029000001, 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:56.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:22:56.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:22:56.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:22:56.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:22:56.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:56.700 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:56.700 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:56.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:56.702 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:56.702+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:56.802 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:56.803+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:56.804 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:56.805 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:56.805 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:56.805 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:56.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:56.903 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:56.904+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:56.915 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:56.915 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:56.915 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:56.915 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:56.915 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:56.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:57.004 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:57.005+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:57.069 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414978003000002, 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:57.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:57.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414956879000002, 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.074 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:57.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:57.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414927612, 2024-04-23T21:22:57.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:57.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.084 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414906849999994, 2024-04-23T21:22:57.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.086 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:57.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:57.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:57.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:57.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:57.093 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414887824000004, 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.094 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:57.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:57.099 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414868984999998, 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:57.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414835582999999, 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:57.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:57.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.109 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414815837000006, 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.110 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:57.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:57.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:57.117 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.118 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.118 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.118 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414797734000004, 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:57.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:57.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:57.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414778533000003, 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.123 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:57.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.127 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414753133999994, 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.128 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:57.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414728009000001, 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.132 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:57.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.136 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414702997999996, 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.141 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414679429000003, 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.142 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414656273999995, 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:22:57.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:22:57.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414635531000002, 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.151 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:22:57.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414617230000005, 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:22:57.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.41459965, 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:22:57.159 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:22:57.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 65.414570127000005, 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:22:57.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:22:57.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 56.675315062999999, 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:22:57.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:57.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:22:57.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:22:57.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:22:57.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:57.173 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:57.174 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:22:57.175 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:57.176+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:57.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:22:57.275 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:57.276+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:57.292 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:57.292 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:57.292 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:57.292 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:57.302 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:22:57.312 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:57.312 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:57.330 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:57.331 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:57.348 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:57.348 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:57.365 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:57.365 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:57.376 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:57.377+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:57.383 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:57.383 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:57.383 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:22:57.400 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:57.401 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:57.419 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:57.419 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:22:57.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:22:57.448 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:57.448 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:57.457 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:22:57.477 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:57.479+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:57.558 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:57.558 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:57.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:22:57.611 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:57.612+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:57.671 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:57.671 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:57.680 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:22:57.680 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:57.680+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:57.780 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:57.781+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:57.822 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:57.822 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:57.830 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:22:57.881 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:57.882+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:57.932 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:57.932 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:57.940 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:22:57.982 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:57.983+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:58.043 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.043 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:22:58.095 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:58.096+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:58.153 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.153 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.162 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:22:58.183 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:58.184+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:58.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:22:58.285 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:22:58.285+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:22:58.315 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.316 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:22:58.385 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:22:58.386+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:22:58.434 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.435 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.444 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:22:58.445 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:22:58.445 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:22:58.445 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-04-23T21:22:58.446 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:22:58.487 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:58.488+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:58.587 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:58.587+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:58.590 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.590 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:22:58.689 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:58.690+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:58.703 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.703 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.712 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 114, 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:22:58.238907+0000", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:22:49.185080+0000", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:22:29.344375+0000", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5806080, 2024-04-23T21:22:58.781 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "nodeep-scrub", 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "noscrub", 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 5, 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:22:58.782 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "luminous", 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:22:58.783 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:22:58.784 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:22:58.785 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:58.786 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:58.787 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:58.788 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 4, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 4, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.75, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 3, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 3, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:22:58.789 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:58.790 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:22:58.791 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:58.792 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:58.793 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:58.794 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 8, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 8, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.25, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.795 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:22:58.796 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.797 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "114", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 51, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 114, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:22:58.798 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.799 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.800 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 2.25, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 2.25, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.75, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.690000057220459, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.690000057220459, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.88085174560546875, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.87999999523162842, 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:22:58.801 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.04681396484375, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 105, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.802 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.803 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.804 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.805 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.806 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.807 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 13, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 17, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 45, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 89, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.808 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.809 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.810 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.811 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/723341542", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/723341542", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/723341542", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/723341542", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.812 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 105, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.814 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.815 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.816 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:22:58.817 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 0, 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 0, 2024-04-23T21:22:58.818 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.819 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.820 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.821 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.822 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.823 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 105, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.824 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.825 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.826 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.827 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:22:58.828 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 0, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.829 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 105, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.830 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.831 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.832 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.833 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.834 INFO:teuthology.orchestra.run.smithi111.stdout: "exists" 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 105, 2024-04-23T21:22:58.835 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.836 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.837 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.838 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:22:58.839 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:22:58.840 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 105, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.841 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.842 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.843 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:22:58.844 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:22:58.845 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.846 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.847 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 65536, 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:22:58.848 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 105 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.850 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.851 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [ 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.6", 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "mappings": [ 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "from": 6, 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "to": 7 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [], 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:22:58.852 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [ 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.853 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 32, 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 9 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [ 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 32, 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 1 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.854 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [], 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:22:58.855 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:22:58.857 INFO:tasks.thrashosds.thrasher:j is {'epoch': 114, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:22:58.238907+0000', 'last_up_change': '2024-04-23T21:22:49.185080+0000', 'last_in_change': '2024-04-23T21:22:29.344375+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.75, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 51, 'snap_epoch': 114, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8808517456054688, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.04681396484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 17, 'up_from': 45, 'up_thru': 89, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 723341542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 723341542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 723341542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 723341542}]}, 'public_addr': '172.21.15.111:6817/723341542', 'cluster_addr': '172.21.15.111:6819/723341542', 'heartbeat_back_addr': '172.21.15.111:6823/723341542', 'heartbeat_front_addr': '172.21.15.111:6821/723341542', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 105, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 105}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '1.6', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 9}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:22:58.858 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:22:58.859 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:58.857+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:58.859 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:58.859 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:58.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:22:58.958 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:58.958+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:59.012 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:59.021 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:22:59.058 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:59.059+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:59.126 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:59.126 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:59.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:22:59.155 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:22:59.155 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:22:59.167 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-04-23T21:22:58.243807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1726250916,"num_objects":1559,"num_object_clones":262,"num_object_copies":3118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":819,"num_read":181925,"num_read_kb":1014890,"num_write":18746,"num_write_kb":600256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16061,"ondisk_log_size":16061,"up":71,"acting":71,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":379208,"kb_used_data":71760,"kb_used_omap":15,"kb_used_meta":306928,"kb_avail":660223672,"statfs":{"total":676457349120,"available":676069040128,"internally_reserved":0,"allocated":73482240,"data_stored":601451261,"data_compressed":36788474,"data_compressed_allocated":40783872,"data_compressed_original":577017295,"omap_allocated":16311,"internal_metadata":314294345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-44084121,"num_objects":-26,"num_object_clones":-9,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":-6,"num_read":65,"num_read_kb":49,"num_write":37,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025749"},"pg_stats":[{"pgid":"3.1f","version":"107'476","reported_seq":4498,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.261693+0000","last_change":"2024-04-23T21:22:50.201458+0000","last_active":"2024-04-23T21:22:53.261693+0000","last_peered":"2024-04-23T21:22:53.261693+0000","last_clean":"2024-04-23T21:22:48.318906+0000","last_became_active":"2024-04-23T21:22:50.201458+0000","last_became_peered":"2024-04-23T21:22:50.201458+0000","last_unstale":"2024-04-23T21:22:53.261693+0000","last_undegraded":"2024-04-23T21:22:50.200864+0000","last_fullsized":"2024-04-23T21:22:50.200766+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:03:32.485280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081897000000000003,"stat_sum":{"num_bytes":68560368,"num_objects":41,"num_object_clones":7,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":25,"num_read":6769,"num_read_kb":17445,"num_write":516,"num_write_kb":33867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":41}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.1e","version":"108'410","reported_seq":6750,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.283729+0000","last_change":"2024-04-23T21:22:53.284118+0000","last_active":"2024-04-23T21:22:53.283729+0000","last_peered":"2024-04-23T21:22:53.283729+0000","last_clean":"2024-04-23T21:22:53.283729+0000","last_became_active":"2024-04-23T21:22:33.186559+0000","last_became_peered":"2024-04-23T21:22:33.186559+0000","last_unstale":"2024-04-23T21:22:53.283729+0000","last_undegraded":"2024-04-23T21:22:53.283729+0000","last_fullsized":"2024-04-23T21:22:53.283729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:26:18.621589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088493299999999997,"stat_sum":{"num_bytes":55274288,"num_objects":50,"num_object_clones":10,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":26,"num_read":6408,"num_read_kb":47230,"num_write":449,"num_write_kb":24901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.1d","version":"106'300","reported_seq":8281,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.211019+0000","last_change":"2024-04-23T21:22:50.200976+0000","last_active":"2024-04-23T21:22:53.211019+0000","last_peered":"2024-04-23T21:22:53.211019+0000","last_clean":"2024-04-23T21:22:48.222633+0000","last_became_active":"2024-04-23T21:22:50.200976+0000","last_became_peered":"2024-04-23T21:22:50.200976+0000","last_unstale":"2024-04-23T21:22:53.211019+0000","last_undegraded":"2024-04-23T21:22:50.199987+0000","last_fullsized":"2024-04-23T21:22:50.199902+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:04.538093+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0033353929999999999,"stat_sum":{"num_bytes":28529074,"num_objects":48,"num_object_clones":7,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":29,"num_read":8298,"num_read_kb":31837,"num_write":269,"num_write_kb":7752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":48}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.1c","version":"109'429","reported_seq":3121,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.264586+0000","last_change":"2024-04-23T21:22:50.200181+0000","last_active":"2024-04-23T21:22:53.264586+0000","last_peered":"2024-04-23T21:22:53.264586+0000","last_clean":"2024-04-23T21:22:48.240944+0000","last_became_active":"2024-04-23T21:22:50.200181+0000","last_became_peered":"2024-04-23T21:22:50.200181+0000","last_unstale":"2024-04-23T21:22:53.264586+0000","last_undegraded":"2024-04-23T21:22:50.199507+0000","last_fullsized":"2024-04-23T21:22:50.199419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:43.538215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029319350000000001,"stat_sum":{"num_bytes":64577584,"num_objects":63,"num_object_clones":14,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":34,"num_read":3016,"num_read_kb":24774,"num_write":497,"num_write_kb":16868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":63}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.1b","version":"108'372","reported_seq":10269,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.297796+0000","last_change":"2024-04-23T21:22:53.212399+0000","last_active":"2024-04-23T21:22:53.297796+0000","last_peered":"2024-04-23T21:22:53.297796+0000","last_clean":"2024-04-23T21:22:53.297796+0000","last_became_active":"2024-04-23T21:21:32.844314+0000","last_became_peered":"2024-04-23T21:21:32.844314+0000","last_unstale":"2024-04-23T21:22:53.297796+0000","last_undegraded":"2024-04-23T21:22:53.297796+0000","last_fullsized":"2024-04-23T21:22:53.297796+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:13:25.601906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001517497,"stat_sum":{"num_bytes":91250203,"num_objects":52,"num_object_clones":12,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":23,"num_read":11286,"num_read_kb":51700,"num_write":359,"num_write_kb":56488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.1a","version":"108'812","reported_seq":5837,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.303008+0000","last_change":"2024-04-23T21:22:53.211342+0000","last_active":"2024-04-23T21:22:53.303008+0000","last_peered":"2024-04-23T21:22:53.303008+0000","last_clean":"2024-04-23T21:22:53.303008+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:22:53.303008+0000","last_undegraded":"2024-04-23T21:22:53.303008+0000","last_fullsized":"2024-04-23T21:22:53.303008+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":812,"log_dups_size":0,"ondisk_log_size":812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:45:51.157485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079418299999999995,"stat_sum":{"num_bytes":47930447,"num_objects":51,"num_object_clones":8,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":26,"num_read":9375,"num_read_kb":51941,"num_write":1009,"num_write_kb":37910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.19","version":"113'826","reported_seq":2913,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.196653+0000","last_change":"2024-04-23T21:22:57.196897+0000","last_active":"2024-04-23T21:22:57.196653+0000","last_peered":"2024-04-23T21:22:57.196653+0000","last_clean":"2024-04-23T21:22:57.196653+0000","last_became_active":"2024-04-23T21:22:34.185924+0000","last_became_peered":"2024-04-23T21:22:34.185924+0000","last_unstale":"2024-04-23T21:22:57.196653+0000","last_undegraded":"2024-04-23T21:22:57.196653+0000","last_fullsized":"2024-04-23T21:22:57.196653+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:44:14.233299+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0086035250000000008,"stat_sum":{"num_bytes":28178742,"num_objects":44,"num_object_clones":6,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":28,"num_read":5348,"num_read_kb":14810,"num_write":1109,"num_write_kb":9872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"3.18","version":"112'665","reported_seq":5571,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.190350+0000","last_change":"2024-04-23T21:22:57.191714+0000","last_active":"2024-04-23T21:22:57.190350+0000","last_peered":"2024-04-23T21:22:57.190350+0000","last_clean":"2024-04-23T21:22:57.190350+0000","last_became_active":"2024-04-23T21:22:33.189427+0000","last_became_peered":"2024-04-23T21:22:33.189427+0000","last_unstale":"2024-04-23T21:22:57.190350+0000","last_undegraded":"2024-04-23T21:22:57.190350+0000","last_fullsized":"2024-04-23T21:22:57.190350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:03:37.734489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027608020000000001,"stat_sum":{"num_bytes":33767919,"num_objects":57,"num_object_clones":12,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":33,"num_read":7724,"num_read_kb":24867,"num_write":831,"num_write_kb":7573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":5309940,"num_keys_recovered":11,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"3.17","version":"107'527","reported_seq":2751,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.261740+0000","last_change":"2024-04-23T21:22:50.202597+0000","last_active":"2024-04-23T21:22:53.261740+0000","last_peered":"2024-04-23T21:22:53.261740+0000","last_clean":"2024-04-23T21:22:46.501534+0000","last_became_active":"2024-04-23T21:22:50.202597+0000","last_became_peered":"2024-04-23T21:22:50.202597+0000","last_unstale":"2024-04-23T21:22:53.261740+0000","last_undegraded":"2024-04-23T21:22:50.201119+0000","last_fullsized":"2024-04-23T21:22:50.201031+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:49:05.611565+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.037804144999999997,"stat_sum":{"num_bytes":33108195,"num_objects":58,"num_object_clones":8,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":3854,"num_read_kb":30765,"num_write":559,"num_write_kb":28210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":58}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.16","version":"106'437","reported_seq":5257,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.214627+0000","last_change":"2024-04-23T21:22:50.205477+0000","last_active":"2024-04-23T21:22:53.214627+0000","last_peered":"2024-04-23T21:22:53.214627+0000","last_clean":"2024-04-23T21:22:48.279945+0000","last_became_active":"2024-04-23T21:22:50.205477+0000","last_became_peered":"2024-04-23T21:22:50.205477+0000","last_unstale":"2024-04-23T21:22:53.214627+0000","last_undegraded":"2024-04-23T21:22:50.202180+0000","last_fullsized":"2024-04-23T21:22:50.202084+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:40:00.804227+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0027743109999999998,"stat_sum":{"num_bytes":31791147,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":29,"num_read":6184,"num_read_kb":36256,"num_write":464,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":45}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.15","version":"106'344","reported_seq":4515,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.219388+0000","last_change":"2024-04-23T21:22:53.220393+0000","last_active":"2024-04-23T21:22:53.219388+0000","last_peered":"2024-04-23T21:22:53.219388+0000","last_clean":"2024-04-23T21:22:53.219388+0000","last_became_active":"2024-04-23T21:22:33.191679+0000","last_became_peered":"2024-04-23T21:22:33.191679+0000","last_unstale":"2024-04-23T21:22:53.219388+0000","last_undegraded":"2024-04-23T21:22:53.219388+0000","last_fullsized":"2024-04-23T21:22:53.219388+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:28:34.660488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060370529999999997,"stat_sum":{"num_bytes":56277703,"num_objects":62,"num_object_clones":14,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":30,"num_read":4322,"num_read_kb":26762,"num_write":339,"num_write_kb":9614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":5579029,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.14","version":"104'629","reported_seq":5788,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.211853+0000","last_change":"2024-04-23T21:22:53.212213+0000","last_active":"2024-04-23T21:22:53.211853+0000","last_peered":"2024-04-23T21:22:53.211853+0000","last_clean":"2024-04-23T21:22:53.211853+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:22:53.211853+0000","last_undegraded":"2024-04-23T21:22:53.211853+0000","last_fullsized":"2024-04-23T21:22:53.211853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:59:46.553817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014872069999999999,"stat_sum":{"num_bytes":88753165,"num_objects":52,"num_object_clones":8,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":28,"num_read":7642,"num_read_kb":52245,"num_write":764,"num_write_kb":17076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"1.7","version":"0'0","reported_seq":203,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.283149+0000","last_change":"2024-04-23T21:22:30.359764+0000","last_active":"2024-04-23T21:22:53.283149+0000","last_peered":"2024-04-23T21:22:53.283149+0000","last_clean":"2024-04-23T21:22:53.283149+0000","last_became_active":"2024-04-23T21:22:30.358837+0000","last_became_peered":"2024-04-23T21:22:30.358837+0000","last_unstale":"2024-04-23T21:22:53.283149+0000","last_undegraded":"2024-04-23T21:22:53.283149+0000","last_fullsized":"2024-04-23T21:22:53.283149+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:38.092515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"109'431","reported_seq":2727,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.292555+0000","last_change":"2024-04-23T21:22:50.201625+0000","last_active":"2024-04-23T21:22:53.292555+0000","last_peered":"2024-04-23T21:22:53.292555+0000","last_clean":"2024-04-23T21:22:48.222469+0000","last_became_active":"2024-04-23T21:22:50.201625+0000","last_became_peered":"2024-04-23T21:22:50.201625+0000","last_unstale":"2024-04-23T21:22:53.292555+0000","last_undegraded":"2024-04-23T21:22:50.200678+0000","last_fullsized":"2024-04-23T21:22:50.200572+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:04:51.758780+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0049865739999999997,"stat_sum":{"num_bytes":55544539,"num_objects":51,"num_object_clones":11,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":22,"num_read":2628,"num_read_kb":19033,"num_write":497,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":10723491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":51}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"1.6","version":"0'0","reported_seq":108,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.214315+0000","last_change":"2024-04-23T21:22:04.355532+0000","last_active":"2024-04-23T21:22:53.214315+0000","last_peered":"2024-04-23T21:22:53.214315+0000","last_clean":"2024-04-23T21:22:53.214315+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:22:53.214315+0000","last_undegraded":"2024-04-23T21:22:53.214315+0000","last_fullsized":"2024-04-23T21:22:53.214315+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:21:16.796367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"112'424","reported_seq":4246,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.188612+0000","last_change":"2024-04-23T21:22:57.189277+0000","last_active":"2024-04-23T21:22:57.188612+0000","last_peered":"2024-04-23T21:22:57.188612+0000","last_clean":"2024-04-23T21:22:57.188612+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:22:57.188612+0000","last_undegraded":"2024-04-23T21:22:57.188612+0000","last_fullsized":"2024-04-23T21:22:57.188612+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:33.964329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010370609999999999,"stat_sum":{"num_bytes":55220287,"num_objects":33,"num_object_clones":6,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":16,"num_read":5175,"num_read_kb":34978,"num_write":356,"num_write_kb":17883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"1.5","version":"0'0","reported_seq":198,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.210889+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:22:53.210889+0000","last_peered":"2024-04-23T21:22:53.210889+0000","last_clean":"2024-04-23T21:22:53.210889+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:22:53.210889+0000","last_undegraded":"2024-04-23T21:22:53.210889+0000","last_fullsized":"2024-04-23T21:22:53.210889+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"108'404","reported_seq":3619,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.214907+0000","last_change":"2024-04-23T21:22:53.215367+0000","last_active":"2024-04-23T21:22:53.214907+0000","last_peered":"2024-04-23T21:22:53.214907+0000","last_clean":"2024-04-23T21:22:53.214907+0000","last_became_active":"2024-04-23T21:22:33.191620+0000","last_became_peered":"2024-04-23T21:22:33.191620+0000","last_unstale":"2024-04-23T21:22:53.214907+0000","last_undegraded":"2024-04-23T21:22:53.214907+0000","last_fullsized":"2024-04-23T21:22:53.214907+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:59:02.796005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012032,"stat_sum":{"num_bytes":38691917,"num_objects":47,"num_object_clones":5,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":29,"num_read":3760,"num_read_kb":24513,"num_write":462,"num_write_kb":11833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":9761436,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":109,"state":"active+undersized","last_fresh":"2024-04-23T21:22:53.214015+0000","last_change":"2024-04-23T21:22:50.202965+0000","last_active":"2024-04-23T21:22:53.214015+0000","last_peered":"2024-04-23T21:22:53.214015+0000","last_clean":"2024-04-23T21:21:08.710721+0000","last_became_active":"2024-04-23T21:22:50.202965+0000","last_became_peered":"2024-04-23T21:22:50.202965+0000","last_unstale":"2024-04-23T21:22:53.214015+0000","last_undegraded":"2024-04-23T21:22:53.214015+0000","last_fullsized":"2024-04-23T21:22:50.201794+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:41:17.487949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"113'371","reported_seq":4226,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.190288+0000","last_change":"2024-04-23T21:22:57.190288+0000","last_active":"2024-04-23T21:22:57.190288+0000","last_peered":"2024-04-23T21:22:57.190288+0000","last_clean":"2024-04-23T21:22:57.190288+0000","last_became_active":"2024-04-23T21:21:49.938724+0000","last_became_peered":"2024-04-23T21:21:49.938724+0000","last_unstale":"2024-04-23T21:22:57.190288+0000","last_undegraded":"2024-04-23T21:22:57.190288+0000","last_fullsized":"2024-04-23T21:22:57.190288+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:34:45.304392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001636743,"stat_sum":{"num_bytes":56706078,"num_objects":50,"num_object_clones":8,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":32,"num_read":5969,"num_read_kb":33589,"num_write":350,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":609695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"3.1","version":"109'388","reported_seq":8444,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.316490+0000","last_change":"2024-04-23T21:22:53.211793+0000","last_active":"2024-04-23T21:22:53.316490+0000","last_peered":"2024-04-23T21:22:53.316490+0000","last_clean":"2024-04-23T21:22:53.316490+0000","last_became_active":"2024-04-23T21:21:32.843523+0000","last_became_peered":"2024-04-23T21:21:32.843523+0000","last_unstale":"2024-04-23T21:22:53.316490+0000","last_undegraded":"2024-04-23T21:22:53.316490+0000","last_fullsized":"2024-04-23T21:22:53.316490+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:41:13.165103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011170539999999999,"stat_sum":{"num_bytes":120414137,"num_objects":50,"num_object_clones":8,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":23,"num_read":8310,"num_read_kb":76059,"num_write":476,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"2.0","version":"16'117","reported_seq":353,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.213797+0000","last_change":"2024-04-23T21:21:49.938339+0000","last_active":"2024-04-23T21:22:53.213797+0000","last_peered":"2024-04-23T21:22:53.213797+0000","last_clean":"2024-04-23T21:22:53.213797+0000","last_became_active":"2024-04-23T21:21:49.937981+0000","last_became_peered":"2024-04-23T21:21:49.937981+0000","last_unstale":"2024-04-23T21:22:53.213797+0000","last_undegraded":"2024-04-23T21:22:53.213797+0000","last_fullsized":"2024-04-23T21:22:53.213797+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:23:36.429878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":154,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.188034+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:22:57.188034+0000","last_peered":"2024-04-23T21:22:57.188034+0000","last_clean":"2024-04-23T21:22:57.188034+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:22:57.188034+0000","last_undegraded":"2024-04-23T21:22:57.188034+0000","last_fullsized":"2024-04-23T21:22:57.188034+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"106'303","reported_seq":1067,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.196959+0000","last_change":"2024-04-23T21:22:57.197182+0000","last_active":"2024-04-23T21:22:57.196959+0000","last_peered":"2024-04-23T21:22:57.196959+0000","last_clean":"2024-04-23T21:22:57.196959+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:22:57.196959+0000","last_undegraded":"2024-04-23T21:22:57.196959+0000","last_fullsized":"2024-04-23T21:22:57.196959+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:44.510942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078951219999999992,"stat_sum":{"num_bytes":17958446,"num_objects":42,"num_object_clones":6,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":25,"num_read":3349,"num_read_kb":9680,"num_write":330,"num_write_kb":3477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"1.2","version":"0'0","reported_seq":198,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.214129+0000","last_change":"2024-04-23T21:21:09.823836+0000","last_active":"2024-04-23T21:22:53.214129+0000","last_peered":"2024-04-23T21:22:53.214129+0000","last_clean":"2024-04-23T21:22:53.214129+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:22:53.214129+0000","last_undegraded":"2024-04-23T21:22:53.214129+0000","last_fullsized":"2024-04-23T21:22:53.214129+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:32.129948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"109'732","reported_seq":7950,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.295208+0000","last_change":"2024-04-23T21:22:53.212571+0000","last_active":"2024-04-23T21:22:53.295208+0000","last_peered":"2024-04-23T21:22:53.295208+0000","last_clean":"2024-04-23T21:22:53.295208+0000","last_became_active":"2024-04-23T21:21:17.781547+0000","last_became_peered":"2024-04-23T21:21:17.781547+0000","last_unstale":"2024-04-23T21:22:53.295208+0000","last_undegraded":"2024-04-23T21:22:53.295208+0000","last_fullsized":"2024-04-23T21:22:53.295208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:26:48.595796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001506388,"stat_sum":{"num_bytes":74718534,"num_objects":62,"num_object_clones":14,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":28,"num_read":10046,"num_read_kb":57566,"num_write":975,"num_write_kb":33432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+undersized","last_fresh":"2024-04-23T21:22:53.261684+0000","last_change":"2024-04-23T21:22:50.201200+0000","last_active":"2024-04-23T21:22:53.261684+0000","last_peered":"2024-04-23T21:22:53.261684+0000","last_clean":"2024-04-23T21:21:08.710721+0000","last_became_active":"2024-04-23T21:22:50.201200+0000","last_became_peered":"2024-04-23T21:22:50.201200+0000","last_unstale":"2024-04-23T21:22:53.261684+0000","last_undegraded":"2024-04-23T21:22:53.261684+0000","last_fullsized":"2024-04-23T21:22:50.200704+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:17:52.551161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"102'540","reported_seq":4826,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.283289+0000","last_change":"2024-04-23T21:22:50.200496+0000","last_active":"2024-04-23T21:22:53.283289+0000","last_peered":"2024-04-23T21:22:53.283289+0000","last_clean":"2024-04-23T21:22:46.469672+0000","last_became_active":"2024-04-23T21:22:50.200496+0000","last_became_peered":"2024-04-23T21:22:50.200496+0000","last_unstale":"2024-04-23T21:22:53.283289+0000","last_undegraded":"2024-04-23T21:22:50.198932+0000","last_fullsized":"2024-04-23T21:22:50.198824+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:44.378237+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0064004500000000002,"stat_sum":{"num_bytes":51347263,"num_objects":39,"num_object_clones":5,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":20,"num_read":6991,"num_read_kb":14339,"num_write":693,"num_write_kb":8858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":39}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"1.0","version":"0'0","reported_seq":198,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.213877+0000","last_change":"2024-04-23T21:21:09.823742+0000","last_active":"2024-04-23T21:22:53.213877+0000","last_peered":"2024-04-23T21:22:53.213877+0000","last_clean":"2024-04-23T21:22:53.213877+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:22:53.213877+0000","last_undegraded":"2024-04-23T21:22:53.213877+0000","last_fullsized":"2024-04-23T21:22:53.213877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:39:33.964305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"113'270","reported_seq":2652,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-04-23T21:22:57.200073+0000","last_change":"2024-04-23T21:22:57.200343+0000","last_active":"2024-04-23T21:22:57.200073+0000","last_peered":"2024-04-23T21:22:57.200073+0000","last_clean":"2024-04-23T21:22:57.200073+0000","last_became_active":"2024-04-23T21:22:33.190419+0000","last_became_peered":"2024-04-23T21:22:33.190419+0000","last_unstale":"2024-04-23T21:22:57.200073+0000","last_undegraded":"2024-04-23T21:22:57.200073+0000","last_fullsized":"2024-04-23T21:22:57.200073+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:04:24.288209+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011141729,"stat_sum":{"num_bytes":21043996,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":24,"num_read":2161,"num_read_kb":15419,"num_write":262,"num_write_kb":12006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":13465209,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"21","length":"8"}]},{"pgid":"3.9","version":"107'327","reported_seq":7337,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.210943+0000","last_change":"2024-04-23T21:22:53.211455+0000","last_active":"2024-04-23T21:22:53.210943+0000","last_peered":"2024-04-23T21:22:53.210943+0000","last_clean":"2024-04-23T21:22:53.210943+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:22:53.210943+0000","last_undegraded":"2024-04-23T21:22:53.210943+0000","last_fullsized":"2024-04-23T21:22:53.210943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:58:45.151864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083071900000000005,"stat_sum":{"num_bytes":80386717,"num_objects":55,"num_object_clones":13,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":27,"num_read":7054,"num_read_kb":46743,"num_write":346,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.a","version":"109'486","reported_seq":3510,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.285214+0000","last_change":"2024-04-23T21:22:53.285562+0000","last_active":"2024-04-23T21:22:53.285214+0000","last_peered":"2024-04-23T21:22:53.285214+0000","last_clean":"2024-04-23T21:22:53.285214+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:22:53.285214+0000","last_undegraded":"2024-04-23T21:22:53.285214+0000","last_fullsized":"2024-04-23T21:22:53.285214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":486,"log_dups_size":0,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:04:45.651105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0023810469999999999,"stat_sum":{"num_bytes":22901159,"num_objects":46,"num_object_clones":5,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":24,"num_read":5307,"num_read_kb":12693,"num_write":473,"num_write_kb":14975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.b","version":"108'431","reported_seq":2403,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.211498+0000","last_change":"2024-04-23T21:22:53.211950+0000","last_active":"2024-04-23T21:22:53.211498+0000","last_peered":"2024-04-23T21:22:53.211498+0000","last_clean":"2024-04-23T21:22:53.211498+0000","last_became_active":"2024-04-23T21:22:33.192667+0000","last_became_peered":"2024-04-23T21:22:33.192667+0000","last_unstale":"2024-04-23T21:22:53.211498+0000","last_undegraded":"2024-04-23T21:22:53.211498+0000","last_fullsized":"2024-04-23T21:22:53.211498+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:05:17.337127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012401479999999999,"stat_sum":{"num_bytes":47307040,"num_objects":48,"num_object_clones":8,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":24,"num_read":2621,"num_read_kb":25830,"num_write":454,"num_write_kb":14307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":9997927,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.c","version":"103'370","reported_seq":1727,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.283387+0000","last_change":"2024-04-23T21:22:50.200455+0000","last_active":"2024-04-23T21:22:53.283387+0000","last_peered":"2024-04-23T21:22:53.283387+0000","last_clean":"2024-04-23T21:22:47.203136+0000","last_became_active":"2024-04-23T21:22:50.200455+0000","last_became_peered":"2024-04-23T21:22:50.200455+0000","last_unstale":"2024-04-23T21:22:53.283387+0000","last_undegraded":"2024-04-23T21:22:50.198941+0000","last_fullsized":"2024-04-23T21:22:50.198846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:23:08.326069+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.018964399,"stat_sum":{"num_bytes":77649999,"num_objects":52,"num_object_clones":12,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":29,"num_read":1441,"num_read_kb":18659,"num_write":335,"num_write_kb":17730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":52}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.d","version":"109'294","reported_seq":2743,"reported_epoch":109,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:22:53.290320+0000","last_change":"2024-04-23T21:22:50.202993+0000","last_active":"2024-04-23T21:22:53.290320+0000","last_peered":"2024-04-23T21:22:53.290320+0000","last_clean":"2024-04-23T21:22:48.192429+0000","last_became_active":"2024-04-23T21:22:50.202993+0000","last_became_peered":"2024-04-23T21:22:50.202993+0000","last_unstale":"2024-04-23T21:22:53.290320+0000","last_undegraded":"2024-04-23T21:22:50.202286+0000","last_fullsized":"2024-04-23T21:22:50.202200+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:33:24.915825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028173719999999998,"stat_sum":{"num_bytes":40211243,"num_objects":46,"num_object_clones":7,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":27,"num_read":2710,"num_read_kb":11630,"num_write":268,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[{"shards":"7","objects":46}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"29","length":"5"}]},{"pgid":"3.e","version":"108'571","reported_seq":2220,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.220455+0000","last_change":"2024-04-23T21:22:53.220779+0000","last_active":"2024-04-23T21:22:53.220455+0000","last_peered":"2024-04-23T21:22:53.220455+0000","last_clean":"2024-04-23T21:22:53.220455+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:22:53.220455+0000","last_undegraded":"2024-04-23T21:22:53.220455+0000","last_fullsized":"2024-04-23T21:22:53.220455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:07:54.775579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060515880000000001,"stat_sum":{"num_bytes":27930339,"num_objects":38,"num_object_clones":4,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":21,"num_read":3784,"num_read_kb":11493,"num_write":674,"num_write_kb":15955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.f","version":"109'932","reported_seq":10096,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.228583+0000","last_change":"2024-04-23T21:22:53.228583+0000","last_active":"2024-04-23T21:22:53.228583+0000","last_peered":"2024-04-23T21:22:53.228583+0000","last_clean":"2024-04-23T21:22:53.228583+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:22:53.228583+0000","last_undegraded":"2024-04-23T21:22:53.228583+0000","last_fullsized":"2024-04-23T21:22:53.228583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":932,"log_dups_size":0,"ondisk_log_size":932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:57:48.650655+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.014271717999999999,"stat_sum":{"num_bytes":56090883,"num_objects":47,"num_object_clones":4,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":20,"num_read":13173,"num_read_kb":58737,"num_write":1280,"num_write_kb":16409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.10","version":"108'490","reported_seq":3415,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.283352+0000","last_change":"2024-04-23T21:22:53.283660+0000","last_active":"2024-04-23T21:22:53.283352+0000","last_peered":"2024-04-23T21:22:53.283352+0000","last_clean":"2024-04-23T21:22:53.283352+0000","last_became_active":"2024-04-23T21:21:17.779304+0000","last_became_peered":"2024-04-23T21:21:17.779304+0000","last_unstale":"2024-04-23T21:22:53.283352+0000","last_undegraded":"2024-04-23T21:22:53.283352+0000","last_fullsized":"2024-04-23T21:22:53.283352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:34:07.138720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047814400000000001,"stat_sum":{"num_bytes":78316304,"num_objects":42,"num_object_clones":4,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":19,"num_read":4482,"num_read_kb":33821,"num_write":521,"num_write_kb":20706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.11","version":"109'281","reported_seq":2322,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.219014+0000","last_change":"2024-04-23T21:22:53.219330+0000","last_active":"2024-04-23T21:22:53.219014+0000","last_peered":"2024-04-23T21:22:53.219014+0000","last_clean":"2024-04-23T21:22:53.219014+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:22:53.219014+0000","last_undegraded":"2024-04-23T21:22:53.219014+0000","last_fullsized":"2024-04-23T21:22:53.219014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:35:59.046289+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0053963170000000003,"stat_sum":{"num_bytes":56625468,"num_objects":38,"num_object_clones":6,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":16,"num_read":1892,"num_read_kb":20600,"num_write":289,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.12","version":"109'827","reported_seq":3099,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.316039+0000","last_change":"2024-04-23T21:22:53.214787+0000","last_active":"2024-04-23T21:22:53.316039+0000","last_peered":"2024-04-23T21:22:53.316039+0000","last_clean":"2024-04-23T21:22:53.316039+0000","last_became_active":"2024-04-23T21:21:32.839694+0000","last_became_peered":"2024-04-23T21:21:32.839694+0000","last_unstale":"2024-04-23T21:22:53.316039+0000","last_undegraded":"2024-04-23T21:22:53.316039+0000","last_fullsized":"2024-04-23T21:22:53.316039+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:35:25.835789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078530500000000003,"stat_sum":{"num_bytes":50167638,"num_objects":53,"num_object_clones":5,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":23,"num_read":5489,"num_read_kb":39609,"num_write":1113,"num_write_kb":21644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]},{"pgid":"3.13","version":"107'845","reported_seq":2884,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-23T21:22:53.214233+0000","last_change":"2024-04-23T21:22:53.214765+0000","last_active":"2024-04-23T21:22:53.214233+0000","last_peered":"2024-04-23T21:22:53.214233+0000","last_clean":"2024-04-23T21:22:53.214233+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:22:53.214233+0000","last_undegraded":"2024-04-23T21:22:53.214233+0000","last_fullsized":"2024-04-23T21:22:53.214233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:16:24.982225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077937199999999996,"stat_sum":{"num_bytes":68429726,"num_objects":53,"num_object_clones":9,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":25,"num_read":5241,"num_read_kb":39167,"num_write":1268,"num_write_kb":24112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"9"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1725660548,"num_objects":1557,"num_object_clones":262,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":819,"num_read":181809,"num_read_kb":1014790,"num_write":18587,"num_write_kb":597840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":93902434,"num_keys_recovered":189,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":63299584,"data_stored":590904697,"data_compressed":36054681,"data_compressed_allocated":39890944,"data_compressed_original":570991644,"omap_allocated":0,"internal_metadata":0},"log_size":15944,"ondisk_log_size":15944,"up":55,"acting":55,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64920,"kb_used_data":11520,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94306920,"statfs":{"total":96636764160,"available":96570286080,"internally_reserved":0,"allocated":11796480,"data_stored":86168402,"data_compressed":6340246,"data_compressed_allocated":6733824,"data_compressed_original":82409188,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39952,"kb_used_data":4604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":4714496,"data_stored":29210833,"data_compressed":1598659,"data_compressed_allocated":1896448,"data_compressed_original":27375507,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48928,"kb_used_data":9424,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322912,"statfs":{"total":96636764160,"available":96586661888,"internally_reserved":0,"allocated":9650176,"data_stored":82819704,"data_compressed":4663484,"data_compressed_allocated":5410816,"data_compressed_original":79580353,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76788,"kb_used_data":19204,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94295052,"statfs":{"total":96636764160,"available":96558133248,"internally_reserved":0,"allocated":19664896,"data_stored":178978079,"data_compressed":11024689,"data_compressed_allocated":11878400,"data_compressed_original":172777167,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41308,"kb_used_data":5968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330532,"statfs":{"total":96636764160,"available":96594464768,"internally_reserved":0,"allocated":6111232,"data_stored":38453885,"data_compressed":2267641,"data_compressed_allocated":2682880,"data_compressed_original":36055622,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528335,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44044,"kb_used_data":7152,"kb_used_omap":6,"kb_used_meta":36665,"kb_avail":94327796,"statfs":{"total":96636764160,"available":96591663104,"internally_reserved":0,"allocated":7323648,"data_stored":64939016,"data_compressed":3790565,"data_compressed_allocated":3776512,"data_compressed_original":62402234,"omap_allocated":6748,"internal_metadata":37545380},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63268,"kb_used_data":13888,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94308572,"statfs":{"total":96636764160,"available":96571977728,"internally_reserved":0,"allocated":14221312,"data_stored":120881342,"data_compressed":7103190,"data_compressed_allocated":8404992,"data_compressed_original":116417224,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12931072,"data_stored":120269799,"data_compressed":7103190,"data_compressed_allocated":8404992,"data_compressed_original":116417224,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5742592,"data_stored":63491960,"data_compressed":3754186,"data_compressed_allocated":3727360,"data_compressed_original":61763258,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4743168,"data_stored":37717656,"data_compressed":2264918,"data_compressed_allocated":2678784,"data_compressed_original":35998278,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":6676463,"data_compressed":354993,"data_compressed_allocated":483328,"data_compressed_original":6245965,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17440768,"data_stored":172639355,"data_compressed":10516058,"data_compressed_allocated":11239424,"data_compressed_original":167276239,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":82270788,"data_compressed":4663484,"data_compressed_allocated":5410816,"data_compressed_original":79580353,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3416064,"data_stored":28582110,"data_compressed":1598067,"data_compressed_allocated":1892352,"data_compressed_original":27363219,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9523200,"data_stored":79256566,"data_compressed":5799785,"data_compressed_allocated":6053888,"data_compressed_original":76347108,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:22:59.170 INFO:tasks.thrashosds.thrasher:Setting 3.3 to [0, 2, 1, 5] 2024-04-23T21:22:59.170 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.3', '0', '2', '1', '5'] 2024-04-23T21:22:59.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.3 0 2 1 5 2024-04-23T21:22:59.179 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:22:59.180+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:22:59.246 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:22:59.255 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:59.255 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:59.260 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:59.261+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:59.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:22:59.361 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:59.362+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:59.415 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:59.415 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:59.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:22:59.424 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:22:59.466 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:59.467+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:59.527 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:22:59.527 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:22:59.536 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:22:59.543 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:22:59.543 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:22:59.560 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:22:59.560 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:22:59.562 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:22:59.564+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:22:59.577 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:22:59.577 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:22:59.594 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:22:59.594 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:22:59.611 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:22:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:22:59.611 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:22:59.629 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:22:59.629 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:22:59.641 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:59.642 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:59.642 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:22:59.642 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:59.647 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:22:59.647 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:22:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:22:59.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:22:59.652 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:22:59.695 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:22:59.696+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:22:59.754 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:59.754 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:22:59.754 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:22:59.755 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:22:59.755 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:22:59.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:22:59.764 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:22:59.766+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:22:59.866 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:22:59.867+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:22:59.908 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:22:59.908 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:22:59.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253217454999998, 2024-04-23T21:22:59.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.910 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:22:59.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:22:59.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:22:59.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:22:59.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.918 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.919 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.919 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:59.919 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253196330999998, 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.920 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:59.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:22:59.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:22:59.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:22:59.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:22:59.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:22:59.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:22:59.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.928 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.928 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:22:59.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:22:59.929 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.930 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253167063999996, 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.931 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:59.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.935 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:22:59.936 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:22:59.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253146302000005, 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.941 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:22:59.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.947 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.948 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253127276000001, 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.949 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:59.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:22:59.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:22:59.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:22:59.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253108436999995, 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:22:59.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:22:59.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:22:59.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253075034999995, 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.960 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.963 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253055289000002, 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.964 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:22:59.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:22:59.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253037186, 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:22:59.968 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:22:59.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:22:59.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.253017985, 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:22:59.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:22:59.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252992586000005, 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:22:59.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.984 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.985 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:22:59.986 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:22:59.986 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.986 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.986 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.986 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.987 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252967460999997, 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.988 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:22:59.989 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:22:59.990 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:22:59.991 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252942450000006, 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.992 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.993 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.994 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:22:59.995 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252918880999999, 2024-04-23T21:22:59.996 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:22:59.997 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.998 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:22:59.999 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.000 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252895726000006, 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.001 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.002 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.003 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.004 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.005 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252874982999998, 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.006 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.007 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.008 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:00.009 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:00.010 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:00.011 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252856682000001, 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.012 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:00.013 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.015 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.016 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.017 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:00.017 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:00.017 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.017 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.017 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.018 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:00.019 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:00.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.020 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.020 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.020 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.020 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252839101999996, 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.021 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.022 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.022 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.022 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:00.023 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.024 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.025 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:00.025 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:00.025 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.025 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.025 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.026 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:00.027 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:00.028 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.028 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.028 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.028 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:00.028 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:00.029 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.030 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.030 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.030 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:00.030 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:00.030 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:00.031 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:00.032 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.252809579000001, 2024-04-23T21:23:00.032 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:00.032 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.032 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.032 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.033 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.034 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.035 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.036 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:00.037 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:00.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.039 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.039 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.039 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:00.039 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:00.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.040 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 59.513554515000003, 2024-04-23T21:23:00.042 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.043 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.044 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.045 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.046 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.047 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:00.048 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:00.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:00.050 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:00.051 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.052 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:00.053 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:00.054 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:00.054+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:00.155 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:00.155+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:00.157 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:00.157 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:00.157 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:00.157 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:00.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:00.256 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:00.257+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:00.262 INFO:teuthology.orchestra.run.smithi111.stderr:set 3.3 pg_upmap_items mapping to [0->2,1->5] 2024-04-23T21:23:00.316 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:00.317 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:00.317 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:00.317 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:00.317 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:00.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:00.357 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:00.358+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:00.430 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 55.302444571999999, 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:00.431 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:00.432 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:00.433 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:00.434 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:00.435 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502470828, 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.436 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.437 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.438 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.439 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:00.440 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502448168999997, 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.441 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:00.442 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.443 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.444 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502421648999999, 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.445 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.446 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.447 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.448 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.449 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502401585000001, 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.450 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:00.451 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.452 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.453 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502108915000001, 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.454 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.455 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.456 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.457 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.458 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.502099010000002, 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.459 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:00.460 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.461 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:00.462 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501796229999997, 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.463 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:00.464 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.465 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.466 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.467 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501759550000003, 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.468 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.469 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.470 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.471 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501737927000001, 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:00.472 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:00.473 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.474 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.475 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501717681999999, 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.476 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.477 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:00.478 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:00.479 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501516365000001, 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.480 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.481 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.482 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.483 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.484 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.501081694, 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:00.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.486 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:00.487 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.488 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.489 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500814869000003, 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.490 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.491 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.492 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.493 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500792668000003, 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.494 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.495 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:00.496 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.497 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500773078999998, 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:00.498 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.499 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:00.500 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:00.501 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.502 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.503 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.503 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:00.503 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500434767000002, 2024-04-23T21:23:00.505 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:00.505 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.505 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.505 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:00.506 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.507 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.508 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500414184999997, 2024-04-23T21:23:00.509 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.510 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:00.511 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:00.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.513 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500395161, 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.514 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:00.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:00.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:00.517 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 41.500168997000003, 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:00.518 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.519 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.520 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:00.521 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:00.522 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:00.524 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:00.525 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:00.525+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:00.624 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:00.625+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:00.634 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:00.634 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:00.634 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:00.634 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:00.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:00.727 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:00.728+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:00.785 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:00.785 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:00.786 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:00.786 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:00.786 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:00.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:00.836 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:00.837+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:00.926 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:00.927+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.940 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 71.999636234999997, 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170237305999997, 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.945 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:00.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:00.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:00.949 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170214371999997, 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:00.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:00.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170189053000001, 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.954 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:00.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:00.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:00.957 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170170333999998, 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.958 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170127653000002, 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.962 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:00.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:00.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170101512999999, 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:00.966 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:00.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:00.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170083595000001, 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.974 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170052216000002, 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.975 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:00.976 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:00.977 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:00.978 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170035087000002, 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.979 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.980 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:00.981 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.982 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:00.983 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.170006549, 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.984 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.985 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:00.986 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.987 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169963564, 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.988 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.989 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.990 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:00.991 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169936327000002, 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:00.992 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:00.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169912539999999, 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:00.996 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:00.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:00.998 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:00.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169887297999999, 2024-04-23T21:23:01.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:01.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.004 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169868532000002, 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.005 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:01.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.007 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169811572, 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.009 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:01.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:01.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169789838, 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:01.013 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:01.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.017 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 47.169765617000003, 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:01.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:01.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:01.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 27.740496562000001, 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:01.022 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:01.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:01.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.028 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:01.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:01.030 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:01.030+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:01.131 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:01.131+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:01.172 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:01.172 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:01.172 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:01.180 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:01.231 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:01.232+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:01.262 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:23:01.285 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:01.286 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:01.286 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:01.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:01.336 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:01.337+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:01.433 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:01.435+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.442 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.012035984999997, 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:01.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.012013174000003, 2024-04-23T21:23:01.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:01.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011993563999994, 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.452 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:01.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:01.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011974701, 2024-04-23T21:23:01.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:01.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:01.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011955405999998, 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.461 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:01.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:01.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011928845, 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:01.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:01.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011887615000006, 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011853888000005, 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011830478999997, 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.479 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.483 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011797228999995, 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.484 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.487 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011773836000003, 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.488 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:01.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.491 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:01.492 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011753287000005, 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:01.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.496 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.011727527000005, 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.497 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:01.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.010889511000002, 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.501 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.505 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.010866179999994, 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.009978887000003, 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:01.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.513 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.009745307000003, 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.514 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.009628609000004, 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.518 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.009254194999997, 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.522 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:01.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.007897310000004, 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:01.531 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:01.532 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:01.574 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:01.575+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:01.636 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:01.637+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:01.651 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:01.677 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:01.678 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:01.678 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:01.679 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:01.686 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:01.737 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:01.737+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:01.773 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:01.774 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:01.791 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:01.791 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:01.808 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:01.809 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:01.825 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:01.825 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:01.832 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:01.833 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:01.838 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:01.839+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:01.842 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:01.843 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:01.844 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:01.844 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:01.862 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:01.862 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:01.879 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:01.880 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:01.883 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:01.938 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:01.939+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:01.946 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:01.946 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:01.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:02.040 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:02.040+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:02.096 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.097 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:02.140 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:02.141+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:02.207 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.207 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.215 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:02.242 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:02.242+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:02.258 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:23:02.323 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.323 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:02.342 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:02.343+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:02.443 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:02.444+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:02.448 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.448 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:02.543 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:02.544+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:02.565 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.565 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.576 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:02.660 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:02.661+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:02.723 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.723 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:02.745 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:02.747+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:02.847 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:02.848+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:02.875 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.875 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.884 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:02.948 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:02.949+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:02.985 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:02.985 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:02.994 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:03.049 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:03.050+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:03.097 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.097 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:03.190 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:03.191+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:03.250 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.250 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.251 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:03.251+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:03.260 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:03.351 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:03.352+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:03.403 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.403 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:03.452 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:03.453+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:03.515 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.515 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.525 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:03.569 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:03.570+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.634 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.643 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:03.653 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:03.654+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:03.754 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:03.755+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:03.786 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.786 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:03.855 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:03.857+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:03.884 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:03.898 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:03.898 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:03.907 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:03.956 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:03.957+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:04.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:04.005 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:04.023 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:04.023 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:04.041 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:04.041 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:04.055 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:04.055 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:04.055 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.057 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:04.058+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:04.059 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:04.059 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:04.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:04.077 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:04.077 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:04.077 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:04.095 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:04.095 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:04.113 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:04.113 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:04.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:04.159 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:04.160+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:04.169 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:04.169 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:04.170 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:04.260 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:04.260+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:04.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:23:04.336 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:04.336 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904283616000001, 2024-04-23T21:23:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:04.337 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.338 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:04.339 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:04.340 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.341 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904260805000007, 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:04.342 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:04.343 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.344 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.345 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904241194999997, 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.346 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.347 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:04.348 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:04.349 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904222332000003, 2024-04-23T21:23:04.350 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.351 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.409 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:04.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:04.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904203037000002, 2024-04-23T21:23:04.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:04.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:04.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904176476000004, 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904135245999996, 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:04.422 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:04.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.426 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904101518999994, 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:04.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:04.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.90407811, 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.431 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.435 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904044859999999, 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.439 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904021467000007, 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.440 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:04.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.904000917999994, 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.444 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:04.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:04.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:04.448 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.903975157999994, 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:04.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:04.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:04.452 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.903137142000006, 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.453 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.455 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.903113810999997, 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.457 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:04.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.902226518000006, 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.461 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.462 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.901992938000006, 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.465 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:04.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:04.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.901876240000007, 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:04.469 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.470 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:04.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.901501826000001, 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.473 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.476 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 94.900144940999994, 2024-04-23T21:23:04.477 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:04.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.481 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:04.484 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:04.484+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:04.584 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:04.585+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:04.587 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:04.684 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:04.686+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:04.738 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:04.738 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:04.738 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:04.788 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:04.788+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:04.886 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:04.886+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.890 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659435712000004, 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.893 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:04.894 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659406044999997, 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.895 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:04.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.899 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659386636999997, 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.900 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:04.903 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659364828999998, 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.904 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659345225999999, 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.908 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:04.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:04.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659325620999994, 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:04.912 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.914 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659304779999999, 2024-04-23T21:23:04.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:04.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.920 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659275515000004, 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.921 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:04.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:04.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:04.924 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.659244322000006, 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.925 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.851161310999998, 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.929 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:04.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847933092999995, 2024-04-23T21:23:04.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.936 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847910712000001, 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.937 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847891434000005, 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:04.941 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:04.945 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847874000000004, 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:04.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:04.947 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.948 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847856550000003, 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.950 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:04.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.953 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847839175000004, 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.954 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.957 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847821132999997, 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.958 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:04.959 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.847796012000003, 2024-04-23T21:23:04.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:04.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.966 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 76.845676440000005, 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:04.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.969 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.970 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.506053449000007, 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:04.971 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:04.972 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:04.973 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:04.974 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:04.975 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:04.976 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:05.017 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:05.017+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:05.088 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:05.088+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:05.118 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:05.118 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:05.118 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:05.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:05.188 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:05.189+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:05.270 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:05.270 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:05.270 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:05.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:05.280 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:05.281 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:05.281 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T21:23:05.281 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.395488 2024-04-23T21:23:05.281 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.3954881307545579 2024-04-23T21:23:05.289 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:23:05.290 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:05.290+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:05.390 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:05.390+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:05.396 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:05.396 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.756575648999998, 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.397 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.398 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.756543542000003, 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.401 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:05.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:05.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:05.405 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.756520155999993, 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:05.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:05.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.409 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.756499848999994, 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.756480037000003, 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.415 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755408638999995, 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.421 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755375294000004, 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.422 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755343765000006, 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.426 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:05.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.428 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:05.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:05.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:05.431 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755316375999996, 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.434 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:05.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:05.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.437 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755304469999999, 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.438 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.441 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755286225999996, 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.442 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.443 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:05.444 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:05.445 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755276409000004, 2024-04-23T21:23:05.446 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.447 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:05.448 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:05.449 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.450 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755266161999998, 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.451 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:05.452 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:05.453 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.454 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755256510999999, 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.455 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.456 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.457 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:05.458 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.459 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.460 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.461 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.755223012000002, 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.462 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.464 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:05.465 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.466 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.467 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:05.468 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.751597587999996, 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:05.469 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.470 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:05.471 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.472 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.473 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:05.474 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:05.475 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.751579719000006, 2024-04-23T21:23:05.476 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:05.477 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.478 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.479 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:05.480 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:05.481 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.482 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.483 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.751560990000002, 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.484 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.485 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.486 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:05.487 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:05.488 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.489 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:05.490 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.751542561999997, 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.491 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:05.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:05.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.751524657000004, 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:05.498 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.501 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:05.502 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:05.504 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:05.504 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:05.504+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:05.605 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:05.605+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:05.608 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:05.608 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:05.608 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:05.608 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:05.618 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:05.706 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:05.706+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:05.762 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:05.763 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:05.763 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:05.763 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:05.763 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:05.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:05.815 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:05.816+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919636580000002, 2024-04-23T21:23:05.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.880 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919598460000003, 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.881 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:05.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919576061000001, 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.885 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:05.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:05.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:05.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919556665000002, 2024-04-23T21:23:05.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:05.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919537066999993, 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.894 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919516853999994, 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.899 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:05.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:05.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:05.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919484687999997, 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:05.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.907 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:05.908 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.909 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.909 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.909 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919460075000003, 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.910 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:05.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.913 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:05.914 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.915 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:05.917 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919439077000007, 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.918 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:05.919 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.920 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.921 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.922 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919418777999994, 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.923 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.924 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:05.925 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.926 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.927 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.919340989000005, 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.928 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:05.929 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.930 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.931 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:05.932 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:05.933 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:05.934 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.917845615000004, 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:05.935 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.936 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.937 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:05.940 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:05.941 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:05.942 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.917795725000005, 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.943 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:05.944 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.945 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.946 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.917771955999996, 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.947 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.948 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.949 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.950 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.917750949999999, 2024-04-23T21:23:05.951 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.952 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.953 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:05.954 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.955 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.917731344999993, 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.956 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:05.957 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:05.958 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.959 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.960 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.961 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.962 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.901729196000005, 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.963 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:05.964 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.965 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.966 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.967 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.968 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.901710684999998, 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.969 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.970 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:05.971 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:05.972 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.973 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.901690239999994, 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.974 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:05.975 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:05.976 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 73.922613028000001, 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.979 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.980 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.981 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:05.982 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:05.983 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:05.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:05.985 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:05.986+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:06.085 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:06.086+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:06.117 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:06.129 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:06.129 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:06.129 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:06.129 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:06.137 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:06.221 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:06.222+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:06.244 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:06.245 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:06.262 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:06.262 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:06.279 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:06.279 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:06.280 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:06.280 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:06.280 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:06.280 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:06.280 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:06.286 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:06.287+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:06.289 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:06.296 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:06.296 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:06.299 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.5 primary-affinity to 0.395488 (8259182) 2024-04-23T21:23:06.319 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:06.319 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:06.319 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:06.337 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:06.338 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:06.356 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:06.356 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:06.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:06.402 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:06.402+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:06.441 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:06.441 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:06.450 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:06.488 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:06.489+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:06.552 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:06.553 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:06.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:06.605 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:06.606+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:06.664 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:06.664 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:06.673 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:06.690 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:06.690+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:06.791 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:06.791+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:06.814 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:06.815 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:06.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:06.892 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:06.893+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:06.930 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:06.930 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:06.938 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:06.993 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:06.993+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:07.041 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.041 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.050 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:07.093 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:07.094+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:07.192 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.192 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.194 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:07.195+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:07.201 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:07.296 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:07.296+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:07.306 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:23:07.309 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.309 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:07.396 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:07.397+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:07.474 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.474 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:07.498 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:07.498+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:07.587 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.587 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.596 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:07.597 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:07.598+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:07.698 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.698 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.699 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:07.699+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:07.707 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:07.799 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:07.800+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:07.848 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.848 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.857 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:07.902 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:07.903+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:07.960 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:07.969 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:08.003 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:08.004+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:08.071 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:08.071 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:08.079 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:08.123 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:08.123+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:08.181 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:08.181 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:08.190 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:08.205 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:08.206+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:08.305 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:08.306+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:08.307 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:23:08.342 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:08.342 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:08.352 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:08.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:08.436 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:08.436+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:08.461 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:08.461 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:08.470 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:08.506 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:08.507+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:08.513 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:08.513 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:08.529 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:08.529 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:08.546 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:08.546 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:08.562 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:08.562 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:08.573 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:08.573 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:08.580 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:08.581 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:08.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:08.598 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:08.598 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:08.607 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:08.607+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:08.616 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:08.616 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:08.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731626235999997, 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:08.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:08.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731588115999998, 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:08.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:08.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731565716999995, 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:08.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:08.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731546320999996, 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:08.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:08.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731526723000002, 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.705 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:08.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:08.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:08.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731506510000003, 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.709 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731474344000006, 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.713 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:08.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:08.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:08.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731449730999998, 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:08.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:08.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731428733000001, 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731408434000002, 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:08.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:08.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.731330645, 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:08.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:08.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:08.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.729835270999999, 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:08.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:08.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:08.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.729785380999999, 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.741 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:08.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.745 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.729761612000004, 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.749 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.729740606000007, 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:08.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:08.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.729721001000001, 2024-04-23T21:23:08.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:08.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.713718852, 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.759 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:08.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:08.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.713700341000006, 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.763 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:08.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:08.767 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 81.713679896000002, 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.771 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 76.734602683999995, 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:08.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:08.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:08.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:08.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:08.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:08.777 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:08.779 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:08.779 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:08.779+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:08.879 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:08.879+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:08.923 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:08.923 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:08.923 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:08.923 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:08.932 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:08.980 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:08.980+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:09.034 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:09.034 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:09.034 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:09.034 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:09.034 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:09.042 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:09.081 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:09.081+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:09.148 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:09.148 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:09.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:09.148 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:09.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493984721000004, 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.152 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493963597000004, 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.153 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:09.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:23:09.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:23:09.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493934330000002, 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:23:09.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:09.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:23:09.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:23:09.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493913567999996, 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493894542000007, 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.165 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:23:09.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493875703, 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493842301000001, 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.173 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:23:09.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:23:09.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493822554999994, 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.177 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:23:09.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:23:09.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493804452000006, 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:23:09.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:23:09.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:23:09.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493785251000006, 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.186 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:09.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.190 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493759851999997, 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:23:09.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493734727000003, 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.195 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:09.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:23:09.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493709715999998, 2024-04-23T21:23:09.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:23:09.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.203 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493686147000005, 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:23:09.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:23:09.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:09.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.207 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493662991999997, 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493642249000004, 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.212 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:09.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.216 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493623948000007, 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:09.220 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493606368000002, 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.221 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:09.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 77.493576845000007, 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:09.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:09.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:09.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 68.754321781000002, 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.229 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.230 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:09.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.235 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:09.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:09.237 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:09.236+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:09.337 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:09.338+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:09.351 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:09.351 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:09.351 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:09.351 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:09.362 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:09.447 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:09.448+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:09.507 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:09.507 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:09.507 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:09.507 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:09.507 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:09.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:09.540 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:09.540+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:09.641 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:09.641+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 64.533377607000006, 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:09.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:09.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83200 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 565248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.667 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.733403862999999, 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038593568000000002, 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83200 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927418+0000", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927419+0000", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0499999999999999e-07 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927420+0000", 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.784e-06 2024-04-23T21:23:09.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927427+0000", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0399999999999998e-06 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927435+0000", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4540000000000007e-06 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965933+0000", 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038497911000000003 2024-04-23T21:23:09.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.965973+0000", 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0194999999999997e-05 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966012+0000", 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8878999999999998e-05 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.671 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.733381203999997, 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:09.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.733354683999998, 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.676 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:09.680 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.733334620000001, 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:09.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:09.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:09.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.73304195, 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.685 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:09.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:09.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.733032045000002, 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.689 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.732729265000003, 2024-04-23T21:23:09.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.697 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.732692585000002, 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.732670962, 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.702 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:09.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.732650716999999, 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:09.706 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.710 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.7324494, 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:09.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:09.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:09.714 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.732014728999999, 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.715 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731747904000002, 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.719 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:09.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731725703000002, 2024-04-23T21:23:09.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:09.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731706113999998, 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731367802000001, 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:09.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:09.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731347220000004, 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731328196, 2024-04-23T21:23:09.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:09.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:09.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.747 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 50.731102032000003, 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:09.748 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:09.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:09.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:09.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:09.752 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:09.753 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:09.754 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:09.753+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:09.854 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:09.854+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:09.856 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:09.856 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:09.857 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:09.857 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:09.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:09.955 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:09.955+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:10.009 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:10.009 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:10.009 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:10.009 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:10.009 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:10.018 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:10.059 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:10.059+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:10.158 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:10.159+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 81.221364617000006, 2024-04-23T21:23:10.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:10.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:10.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.166 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391965687999999, 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:10.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:10.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:10.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:10.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391942753999999, 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.171 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:10.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391917435000003, 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391898716, 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.179 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:10.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:10.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391856035000004, 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:10.183 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391829895000001, 2024-04-23T21:23:10.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.191 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391811977000003, 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:10.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:10.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:10.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.195 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391780597999997, 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.199 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391763468999997, 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.200 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:10.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:10.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:10.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391734931000002, 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.204 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391691946000002, 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.208 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:10.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:10.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391664708999997, 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.212 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391640922000001, 2024-04-23T21:23:10.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:10.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.220 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391615680000001, 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:10.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391596913999997, 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:10.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:10.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:10.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391539954000002, 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.229 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:10.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391518220000002, 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.233 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:10.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 56.391493998999998, 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 36.962224943999999, 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:10.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:10.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:10.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.247 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:10.248 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:10.259 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:10.260+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:10.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:23:10.359 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:10.359 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:10.359 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:10.359 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:10.364 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:10.359+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:10.368 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:10.461 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:10.461+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:10.523 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:10.523 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:10.523 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:10.532 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:10.562 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:10.563+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:10.621 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:10.663 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:10.664+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:10.679 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:10.679 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24913374800001, 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:10.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:10.684 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.249110937, 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:10.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.249091327, 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:10.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:10.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:10.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24907246399999, 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.694 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.698 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24905316900001, 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:10.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:10.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24902660799999, 2024-04-23T21:23:10.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:10.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.707 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.248985378, 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:10.708 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.712 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.248951651, 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:10.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:10.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:10.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24892824200001, 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:10.717 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:10.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.248894992, 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.722 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:10.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:10.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.248871599, 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.727 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.731 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24885105, 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.732 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:10.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:10.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24882529, 2024-04-23T21:23:10.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:10.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:10.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.247987274, 2024-04-23T21:23:10.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:10.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:10.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.247963943, 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:10.747 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:10.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:10.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24707665, 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.752 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:10.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:10.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:10.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24684307, 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.757 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:10.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:10.763 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.764 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.246726372, 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.765 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:10.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:10.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:10.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:10.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:10.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.774 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:10.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.777 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:10.778 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.24635195800001, 2024-04-23T21:23:10.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:10.778 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.778 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:10.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:10.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:10.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:10.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.786 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.786 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 101.244995073, 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:10.788 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:10.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:10.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.794 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:10.795 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.796 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:10.797 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:10.797 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:10.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:10.800 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:10.800 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:10.801 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:10.801 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:10.801 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:10.801 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:10.802 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:10.802 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:10.802 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:10.798+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:10.809 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:10.809 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:10.809 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:10.828 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:10.828 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:10.846 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:10.846 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:10.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:10.899 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:10.900+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:10.913 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:10.913 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:10.922 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:11.000 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:11.001+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:11.023 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.023 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:11.116 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:11.116+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:11.175 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.175 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:11.202 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:11.203+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:11.303 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:11.304+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:11.317 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:11.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:11.318 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 4, 5, 6, 7] 2024-04-23T21:23:11.318 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-04-23T21:23:11.328 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:23:11.338 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.338 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:11.435 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:11.435+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:11.505 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.505 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.505 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:11.505+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:11.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:11.605 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:11.607+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:11.641 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:23:11.641+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e127 definitely_dead 0 2024-04-23T21:23:11.658 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.658 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:11.706 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:11.707+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:11.772 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.772 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.780 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:11.808 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:11.808+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:11.881 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.881 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:11.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:11.908 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:11.909+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:11.992 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:11.992 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.001 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:12.010 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:12.010+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:12.102 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.102 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.111 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:12.112 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:12.111+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:12.214 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:12.213+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:12.214 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.214 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:12.315 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:12.316+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:12.335 INFO:teuthology.orchestra.run.smithi111.stderr:marked out osd.0. 2024-04-23T21:23:12.370 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.371 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:12.422 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:12.422+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:12.517 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:12.517+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:12.523 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.523 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.532 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:12.618 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:12.619+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:12.634 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.635 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.644 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:12.719 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:12.719+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:12.787 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.787 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.796 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:12.840 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:12.840+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:12.850 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:12.911 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:12.912 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:12.921 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:12.922 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:12.921+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:12.974 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:12.974 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:12.992 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:12.993 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:13.009 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:13.010 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:13.022 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:13.022+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:13.028 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:13.028 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:13.045 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:13.046 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:13.046 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:13.063 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:13.064 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:13.066 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.066 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:13.066 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:13.066 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:13.066 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.075 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:13.082 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:13.082 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:13.085 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:13.123 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:13.124+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.186 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.244554582999996, 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:13.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:13.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.192 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415155654000003, 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:13.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:13.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415132720000003, 2024-04-23T21:23:13.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:13.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:13.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415107401, 2024-04-23T21:23:13.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:13.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:13.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415088681999997, 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:13.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:13.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415046001, 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.211 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:13.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.215 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415019860999998, 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.415001943, 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.220 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:13.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.224 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414970564000001, 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:13.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:13.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414953435000001, 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:13.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.233 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414924896999999, 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:13.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:13.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414881911999998, 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.238 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:13.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:13.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:13.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:13.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.243 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414854675000001, 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:13.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:13.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414830887999997, 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.249 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414805645999998, 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414786880000001, 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:13.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:13.263 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414729919999999, 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414708185999999, 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 59.414683965000002, 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.273 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 39.985414910000003, 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.277 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.278 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.279 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:13.280 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.281 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.282 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.283 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:13.283+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:13.283 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:13.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:23:13.410 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:13.410+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:13.432 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.432 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:13.432 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:13.432 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:13.484 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:13.484+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:13.585 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:13.586+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:13.586 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.586 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:13.586 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:13.586 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:13.586 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.595 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:13.686 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:13.687+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:13.699 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.699 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26949482099999, 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.700 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26947201, 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.704 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:13.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:13.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26945240000001, 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.708 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:13.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:13.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269433537, 2024-04-23T21:23:13.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.716 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269414242, 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:13.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:13.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:13.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269387681, 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269346451, 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269312724, 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.729 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:13.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:13.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:13.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26928931499999, 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:13.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:13.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:13.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:13.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26925606499999, 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:13.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:13.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:13.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:13.740 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269232672, 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.741 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:13.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269212123, 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.745 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.269186363, 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.749 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.268348347, 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.753 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26832501600001, 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:13.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.267437723, 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:13.761 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:13.765 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.267204143, 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:13.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.267087445, 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.770 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.773 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.26671303099999, 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:13.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:13.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:13.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:13.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 104.265356146, 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:13.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:13.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:13.782 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:13.827 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:13.827+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:13.886 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:13.886 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:13.886 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:13.886 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:13.887 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:13.888+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:13.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:13.989 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:13.989+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:14.037 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:14.037 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:14.037 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:14.037 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:14.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:14.130 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:14.130+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:14.190 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:14.190 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:14.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959150109000007, 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.191 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:14.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:14.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:14.194 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959120442, 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:14.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:14.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959101034, 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959079226, 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:14.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959059623000002, 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:14.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959040017999996, 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:14.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:14.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.959019177000002, 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:14.220 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.958989912000007, 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:14.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.224 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 87.958958718999995, 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:14.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.228 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.150875708000001, 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:14.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:14.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.232 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147647489999997, 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:14.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:14.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:14.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147625109000003, 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:14.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147605831000007, 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147588397000007, 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.245 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:14.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.248 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147570947000006, 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:14.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:14.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:14.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:14.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147553572000007, 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:14.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:14.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.258 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147535529999999, 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.259 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.147510409000006, 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.263 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:14.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:14.268 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.145390836999994, 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.269 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:14.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:14.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 73.805767845999995, 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.273 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.274 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:14.275 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:14.276 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.277 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:14.278 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:14.191+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:14.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:14.291 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:14.292+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:14.336 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:23:14.392 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:14.393+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:14.452 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:14.452 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:14.452 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:14.452 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:14.462 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:14.493 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:14.494+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:14.579 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:14.579 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:14.579 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:14.580 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:14.580 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:14.588 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:14.594 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:14.595+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:14.694 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:14.695+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.069731113000003, 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.710 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:14.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:14.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:14.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.069699005999993, 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:14.714 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.718 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.069675619999998, 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:14.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:14.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.723 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.069655312999998, 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:14.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.069635500999993, 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:14.732 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068564103, 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:14.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:14.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:14.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.736 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068530757999994, 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:14.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:14.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068499228999997, 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:14.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:14.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068471840000001, 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068459934000003, 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.749 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.750 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.751 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:14.752 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.06844169, 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:14.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.756 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068431872999994, 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:14.757 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.758 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:14.759 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068421626000003, 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.761 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:14.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:14.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068411975000004, 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.765 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:14.766 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.767 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.768 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.068378476000007, 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:14.769 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:14.771 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:14.773 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.064753052, 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:14.774 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:14.775 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.776 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.064735182999996, 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.778 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:14.779 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:14.780 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.781 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.064716454000006, 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:14.782 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:14.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.785 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.064698026000002, 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.786 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.787 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.788 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.789 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 88.064680120999995, 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:14.790 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:14.791 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:14.792 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.793 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.794 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:14.794 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:14.794 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:14.795 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:14.796 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:14.796+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:14.896 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:14.896+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:14.909 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:14.910 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:14.910 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:14.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:14.997 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:14.997+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:15.020 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:15.020 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:15.020 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:15.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:15.086 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:15.098 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:15.098+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:15.133 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:15.133 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:15.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:15.133 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:15.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176989699000003, 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:15.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176951579000004, 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:15.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176929180000002, 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.142 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.143 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:15.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176909784000003, 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.146 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:15.150 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176890185999994, 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:15.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:15.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.154 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176869972999995, 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:15.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:15.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176837806999998, 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:15.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:15.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:15.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176813194000005, 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:15.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:15.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176792195999994, 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.167 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:15.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176771896999995, 2024-04-23T21:23:15.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:15.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:15.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.175 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.176694108000007, 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.179 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.175198734000006, 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.180 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.175148844000006, 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.184 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:15.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:15.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.175125074999997, 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.188 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:15.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:15.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:15.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.175104069, 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:15.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:15.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:15.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:15.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.175084463999994, 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.159082315000006, 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.200 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:15.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:15.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.159063803999999, 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.204 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:15.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 88.159043358999995, 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:15.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 83.179966147000002, 2024-04-23T21:23:15.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:15.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.217 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.218 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.218 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:15.218 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:15.218 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:15.219 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:15.219 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:15.219 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:15.219+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:15.220 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:15.222 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:15.223 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:15.239 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:15.239 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:15.255 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:15.255 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:15.273 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:15.274 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:15.274 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:15.291 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:15.291 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:15.308 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:15.308 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:15.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:15.353 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:15.353+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:15.370 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:15.370 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:15.379 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:15.421 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:15.421+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:15.482 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:15.482 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:15.491 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:15.534 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:15.535+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:15.592 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:15.593 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:15.601 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:15.642 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:15.643+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:15.723 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:15.723+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:15.744 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:15.744 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:15.753 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:15.824 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:15.825+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:15.856 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:15.856 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:15.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:15.925 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:15.925+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:16.008 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.008 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.017 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:16.025 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:16.026+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:16.125 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.125 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.126 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:16.127+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:16.134 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:16.227 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:16.228+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:16.276 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.276 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.285 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:16.329 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:16.329+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:16.363 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:23:16.408 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.409 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.419 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:16.429 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:16.429+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:16.527 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.527 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.529 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:16.530+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:16.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:16.631 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:16.631+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:16.680 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.681 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.689 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:16.733 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:16.733+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:16.791 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.791 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.800 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:16.841 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:16.841+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:16.933 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:16.933+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:16.942 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:16.943 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:16.951 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:17.035 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:17.035+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:17.094 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:17.094 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:17.102 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:17.144 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:17.144+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:17.236 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:17.238+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:17.244 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:17.245 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:17.254 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:17.312 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:17.354 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7] out_osds: [3, 0] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:17.355 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:17.355 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [1, 2, 4, 5, 6, 7] 2024-04-23T21:23:17.355 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-04-23T21:23:17.356 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:17.356+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:17.358 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:17.359 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:17.363 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:23:17.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:17.445 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:17.446 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:17.446 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:17.446+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:17.468 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:17.468 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:17.495 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:17.495 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:17.523 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:17.523 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:17.538 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:17.538 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:17.538 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:17.538 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:17.538 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:17.539 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:17.539+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:17.540 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:17.540 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:17.541 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:17.547 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:17.558 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:17.558 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:17.577 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:17.577 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:17.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:17.640 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:17.641+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:17.665 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:17.665 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.024907913999996, 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.666 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.024875807000001, 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.670 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.024852421000006, 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.674 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:17.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:17.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.024832114000006, 2024-04-23T21:23:17.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:17.682 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.024812302000001, 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.686 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023740903999993, 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.690 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023707559000002, 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:17.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:17.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:17.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.694 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023676030000004, 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:17.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:17.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:17.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023648640999994, 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:17.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023636734999997, 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:17.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023618490999993, 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.707 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:17.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023608674000002, 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.711 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023598426999996, 2024-04-23T21:23:17.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.719 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023588775999997, 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.723 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.023555277, 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:17.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:17.727 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.019929852999994, 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.728 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:17.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:17.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:17.731 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.019911984000004, 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.732 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.019893254999999, 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.736 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:17.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:17.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.019874826999995, 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.740 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.741 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.742 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:17.743 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 91.019856922000002, 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:17.744 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:17.745 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:17.746 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:17.747 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:17.748 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:17.750 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:17.750 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:23:17.694+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e133 definitely_dead 0 2024-04-23T21:23:17.751 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:17.751+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:17.851 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:17.851+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:17.853 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:17.853 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:17.854 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:17.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:17.952 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:17.953+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:18.004 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.004 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:18.004 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:18.004 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:18.005 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.013 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:18.053 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:18.054+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:18.155 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:18.156+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200212738999994, 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:18.157 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:18.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:18.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:18.162 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200174618999995, 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:18.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.166 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200152220000007, 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:18.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200132823999994, 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:18.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:18.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.174 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200113225999999, 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:18.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:18.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200093013, 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.182 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200060847000003, 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.183 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200036233999995, 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.187 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.188 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:18.189 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.190 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.191 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.200015235999999, 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.192 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.193 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.194 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.195 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.199994937, 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.196 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.197 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:18.198 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:18.199 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.199917147999997, 2024-04-23T21:23:18.200 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.201 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:18.202 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.203 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.198421773999996, 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:18.204 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.205 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.206 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.207 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.198371883999997, 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.208 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.209 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:18.210 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.211 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.212 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.198348115000002, 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.213 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:18.214 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.215 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.216 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.198327109000004, 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.217 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:18.218 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:18.219 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:18.220 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.198307503999999, 2024-04-23T21:23:18.221 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.222 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:18.223 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:18.224 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.225 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.182305354999997, 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.226 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:18.227 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:18.228 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.229 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.182286844000004, 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.230 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.231 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.232 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.233 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 91.182266399, 2024-04-23T21:23:18.234 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.235 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:18.236 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:18.237 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.203189187000007, 2024-04-23T21:23:18.238 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.239 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.240 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:18.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:18.243 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.244 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.245 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:18.289 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:18.290+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:18.348 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.348 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:18.348 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:18.348 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:18.358 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:18.358+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:18.376 INFO:teuthology.orchestra.run.smithi111.stderr:marked out osd.7. 2024-04-23T21:23:18.458 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:18.459+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:18.500 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.500 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:18.500 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:18.500 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:18.500 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:18.562 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:18.560+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:18.653 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998819367999999, 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:23:18.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998798244, 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:23:18.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:23:18.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998768976999997, 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:23:18.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998748215000006, 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.666 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:18.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:23:18.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.670 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998729189000002, 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:18.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:23:18.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:23:18.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.674 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998710349999996, 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:23:18.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:23:18.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.678 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998676947999996, 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.682 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998657202000004, 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.683 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:23:18.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998639099000002, 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:23:18.687 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:23:18.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.691 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998619898000001, 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998594499000006, 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.696 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.697 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.699 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998569373999999, 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.700 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:23:18.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:23:18.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998544362999993, 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:23:18.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:23:18.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998520794000001, 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.708 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998497638999993, 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.712 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:18.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998476896, 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.716 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:18.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:18.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998458595000002, 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.720 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:18.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:18.724 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998441014999997, 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:18.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:18.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:18.728 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.998411492000002, 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:18.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.259156427999997, 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:18.733 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:18.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:18.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:18.738 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.740 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:18.741 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:18.740+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:18.841 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:18.841+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:18.843 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.844 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:18.844 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:18.844 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.852 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:18.942 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:18.943+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:18.955 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:18.956 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:18.956 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:18.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:19.043 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:19.043+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 73.940557525000003, 2024-04-23T21:23:19.068 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:19.069 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.070 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:19.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:19.072 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:19.073 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:19.074 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.140561122000001, 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.075 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.076 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.077 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:19.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.140534602000002, 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.079 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.080 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:19.081 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.082 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.083 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.140514537999998, 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.084 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:19.085 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.086 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:19.087 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:19.088 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.089 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:19.090 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:19.091 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.140221867999998, 2024-04-23T21:23:19.092 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.093 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.094 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:19.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.140211962999999, 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:19.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139909183, 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.103 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:19.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:19.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139872502999999, 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:19.107 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:19.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:19.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139850879999997, 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.112 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:19.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.114 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:19.116 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139830635000003, 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.117 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.120 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139629317999997, 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.122 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.126 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.139194646999997, 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.127 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:19.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:19.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138927821999999, 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.131 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:19.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:19.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138905620999999, 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:19.135 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:19.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:19.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:19.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:19.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138886032000002, 2024-04-23T21:23:19.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.144 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:19.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:19.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.150 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138547719999998, 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.152 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:19.154 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138527138000001, 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.155 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:19.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:19.159 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.138508113999997, 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:19.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 60.13828195, 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:19.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:19.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:140851 3.12 3:4c3ea8ab:::rbd_header.11189a87d8f4:head [watch reconnect cookie 140596427927216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 5.7252001149999998, 2024-04-23T21:23:19.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0289291229999999, 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 140851 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342804+0000", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.576e-06 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342809+0000", 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5969999999999999e-06 2024-04-23T21:23:19.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342821+0000", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1286e-05 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342830+0000", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2560000000000008e-06 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342904+0000", 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4219000000000002e-05 2024-04-23T21:23:19.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342915+0000", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1090999999999999e-05 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364226+0000", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0213109119999999 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364289+0000", 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3180000000000002e-05 2024-04-23T21:23:19.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366791+0000", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025013290000000001 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366909+0000", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011882799999999999 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.371731+0000", 2024-04-23T21:23:19.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048218489999999996 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:19.174 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:19.176 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:19.176 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:19.176+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:19.276 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:19.277+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:141344 3.12 3.15017db2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:23:18.398550+0000", 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.89277463700000004, 2024-04-23T21:23:19.290 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.89279566399999999, 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "waiting for peered", 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 141344 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398550+0000", 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.291 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398550+0000", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398552+0000", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4109999999999999e-06 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398559+0000", 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3320000000000003e-06 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.292 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398573+0000", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3964999999999999e-05 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398582+0000", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6289999999999999e-06 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398646+0000", 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4280000000000001e-05 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.293 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:18.398655+0000", 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8939999999999999e-06 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 1 2024-04-23T21:23:19.294 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:19.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:19.377 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:19.377+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:19.384 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:23:19.407 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:19.407 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:19.407 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:19.407 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:19.407 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:19.417 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:19.502 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:19.502+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 90.62370919, 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:19.565 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:19.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:19.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:19.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794310261000007, 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794287327000006, 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.574 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:19.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:19.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794262008000004, 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:19.578 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794243288999994, 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.583 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:19.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:19.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794200607999997, 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:19.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:19.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:19.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:19.590 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794174467999994, 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.591 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794156549999997, 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794125171000005, 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.599 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:19.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:19.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:19.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794108042000005, 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.603 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794079503999995, 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:19.607 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794036519000002, 2024-04-23T21:23:19.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.612 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.613 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.614 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.794009282000005, 2024-04-23T21:23:19.615 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.616 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.617 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.618 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.619 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793985495000001, 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:19.620 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:19.621 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:19.622 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.623 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793960252999995, 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.624 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.625 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.626 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.627 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793941486999998, 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.628 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.629 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:19.630 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.631 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:19.632 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793884527000003, 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:19.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:19.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.636 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793862793000002, 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 65.793838571999999, 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.644 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 46.364569517, 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.645 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:19.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:19.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:19.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:19.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:19.651 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:19.652 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:19.653 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:19.654+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:19.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:19.753 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:19.753+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:19.801 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:19.801 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:19.801 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:19.810 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:19.815 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:19.815 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:19.835 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:19.835 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:19.852 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:19.853 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:19.853 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:19.854+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:19.869 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:19.870 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:19.887 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:19.888 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:19.888 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:19.905 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:19.905 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:19.920 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:19.920 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:19.923 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:19.923 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:19.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:19.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:19.954 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:19.955+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:20.055 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:20.056+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.071 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.080 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:20.164 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:20.164+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:20.222 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.223 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.232 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:20.258 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:20.259+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:20.359 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:20.359+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:20.375 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.375 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.384 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:20.389 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:23:20.459 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:20.460+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:20.517 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.517 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:20.560 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:20.561+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:20.655 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.655 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.660 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:20.661+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:20.666 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:20.762 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:20.762+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:20.809 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.809 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.819 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:20.862 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:20.863+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:20.927 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:20.928 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:20.937 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:20.963 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:20.963+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:21.044 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.044 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:21.063 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:21.064+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:21.164 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:21.165+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:21.169 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.169 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.179 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:21.265 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:21.265+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:21.328 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.328 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.338 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:21.366 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:21.366+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:21.456 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.457 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.466 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:21.467 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:21.467+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:21.568 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.568 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.569 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:21.569+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:21.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:21.670 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:21.670+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:21.679 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.679 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.688 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:21.771 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:21.772+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:21.831 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.831 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.840 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:21.885 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:21.885+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:21.928 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:21.943 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:21.943 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:21.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:21.993 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:21.993+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:22.052 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:22.052 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:22.069 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:22.069 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:22.074 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:22.074+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:22.085 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:22.086 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:22.094 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:22.095 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:22.102 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:22.102 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:22.103 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:22.119 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:22.120 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:22.120 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:22.137 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:22.137 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:22.155 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:22.155 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:22.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:22.174 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:22.176+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:22.214 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.214 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:22.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.214 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:141514 3.18 3:18c60915:::rbd_header.11181e0f6ea:head [call rbd.get_data_pool in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e138)", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:23:22.215017+0000", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 0.00013149, 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015497699999999999, 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 141514 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215017+0000", 2024-04-23T21:23:22.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215017+0000", 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215018+0000", 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8199999999999999e-07 2024-04-23T21:23:22.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215022+0000", 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0470000000000004e-06 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215034+0000", 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2257e-05 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215043+0000", 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1339999999999993e-06 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215095+0000", 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1700999999999998e-05 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:23:22.215176+0000", 2024-04-23T21:23:22.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1409999999999995e-05 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 1 2024-04-23T21:23:22.219 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:22.225 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:22.275 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:22.276+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:22.335 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.335 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:22.335 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:22.335 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:22.335 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:22.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:22.377 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:22.378+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:22.411 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:23:22.478 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:22.479+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 93.550199214000003, 2024-04-23T21:23:22.491 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.492 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.493 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.494 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.495 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720800284999996, 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.496 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:22.497 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:22.498 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:22.499 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.500 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720777350999995, 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:22.501 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.502 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.503 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.504 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720752032000007, 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.505 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:22.506 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:22.507 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.508 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.509 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720733312999997, 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.510 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.511 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.512 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.513 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720690632, 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.514 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.515 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:22.516 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:22.517 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720664491999997, 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.518 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.519 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.520 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.521 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.522 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720646574, 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.523 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.524 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:22.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720615194999993, 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.529 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720598065999994, 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:22.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.536 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720569527999999, 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:22.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:22.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720526543000005, 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.541 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:22.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720499305999994, 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:22.545 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:22.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:22.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:22.549 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720475519000004, 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.550 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.554 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720450276999998, 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.555 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:22.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:22.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:22.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720431511000001, 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.559 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.563 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720374551000006, 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:22.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720352817000006, 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.568 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:22.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:22.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 68.720328596000002, 2024-04-23T21:23:22.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.574 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:22.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.577 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 49.291059541000003, 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:22.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:22.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:22.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:22.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:22.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:22.583 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.584 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.584 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.584 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.584 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:22.585 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:22.585+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:22.585 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:22.686 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:22.687+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:22.730 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.730 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:22.730 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:22.730 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:22.739 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:22.787 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:22.787+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:22.841 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.841 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:22.841 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:22.841 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:22.841 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:22.849 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:22.887 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:22.888+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:22.988 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:22.990+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:22.993 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56304925000001, 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:22.994 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:22.995 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:22.996 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:22.997 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.563026439, 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:22.998 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:22.999 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:23.000 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.001 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.563006829, 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.002 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.003 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.004 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.005 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.006 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56298796599999, 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.007 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.008 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.009 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:23.010 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56296867099999, 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.011 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.012 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.013 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:23.014 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.015 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56294210999999, 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.016 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:23.017 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.018 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.019 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56290088, 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.020 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.021 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.022 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:23.023 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.024 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.562867153, 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.025 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:23.026 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:23.027 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:23.028 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56284374400001, 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.029 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.030 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.031 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:23.032 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.033 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.562810494, 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.034 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:23.035 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:23.036 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:23.037 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.562787101, 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.038 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:23.039 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.040 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.041 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.042 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.562766552, 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.043 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.044 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:23.045 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.046 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.047 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:23.048 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.562740792, 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.050 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:23.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:23.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.561902776, 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.057 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:23.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:23.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:23.063 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.063 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.063 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.561879445, 2024-04-23T21:23:23.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.069 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.560992152, 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.071 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:23.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.073 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:23.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.560758572, 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:23.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:23.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:23.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.083 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.560641874, 2024-04-23T21:23:23.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.085 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:23.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:23.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:23.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.56026746000001, 2024-04-23T21:23:23.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.094 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:23.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:23.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 113.558910575, 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:23.099 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:23.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:23.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:23.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.103 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.104 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.104 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.104 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.104 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.105 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:23.106 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:23.106+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:23.189 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:23:23.206 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:23.206+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:23.213 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.213 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:23.213 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:23.213 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.222 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:23.307 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:23.307+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:23.367 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.367 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:23.367 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:23.367 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:23.367 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.376 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:23.398 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6] out_osds: [3, 0, 7] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:23.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:23.398 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-23T21:23:23.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-04-23T21:23:23.417 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:23.417+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:23.508 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:23.509+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:23.523 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.523 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292838383000003, 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:23.524 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.525 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:23.526 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:23.527 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.528 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292808715999996, 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.529 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.530 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.531 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.533 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292789307999996, 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.534 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.535 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.536 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292767499999997, 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:23.538 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:23.539 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.541 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.542 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292747896999998, 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:23.544 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.545 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.546 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292728292000007, 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:23.547 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:23.548 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.550 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.551 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292707450999998, 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:23.552 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:23.553 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:23.554 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:23.555 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292678186000003, 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.556 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.557 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.558 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.559 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:23.560 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.292646993000005, 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.561 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.562 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.563 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.564 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.484563981999997, 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.565 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.566 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.567 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.568 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.569 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481335763999994, 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.570 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.571 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:23.572 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.573 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481313383, 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.574 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.575 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.576 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.577 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481294105000003, 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:23.578 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:23.579 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.580 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.581 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481276671000003, 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.582 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.583 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:23.584 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:23.585 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481259221000002, 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.586 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.587 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.588 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.589 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:23.590 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481241846000003, 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.591 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:23.592 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:23.593 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.594 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481223803999995, 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.595 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.596 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.597 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.598 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.481198683000002, 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.599 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:23.600 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:23.601 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:23.602 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.479079111000004, 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:23.603 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:23.604 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:23.605 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:23.606 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 83.139456120000006, 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.607 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.608 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.609 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.610 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.611 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.613 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:23.613 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:23.613+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:23.707 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "epoch": 139, 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "fsid": "3556a001-ee44-4b6e-8d50-1bc45ce74758", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "created": "2024-04-23T21:20:57.391123+0000", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "modified": "2024-04-23T21:23:23.189206+0000", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "last_up_change": "2024-04-23T21:22:49.185080+0000", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "last_in_change": "2024-04-23T21:23:18.371965+0000", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_num": 5804032, 2024-04-23T21:23:23.708 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_set": [ 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "nodeep-scrub", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "pglog_hardlimit", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "purged_snapdirs", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "recovery_deletes", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "sortbitwise" 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_version": 5, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "full_ratio": 0.94999998807907104, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_snapshot": "", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_max": 3, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "max_osd": 8, 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "require_min_compat_client": "luminous", 2024-04-23T21:23:23.709 INFO:teuthology.orchestra.run.smithi111.stdout: "min_compat_client": "jewel", 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "require_osd_release": "squid", 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "allow_crimson": false, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pools": [ 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 1, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "rbd", 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:08.380744+0000", 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:23:23.710 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 8, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 8, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 8, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 8, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 8, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:23:23.711 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "16", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:23:23.712 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:23:23.713 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:23:23.714 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 4, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 4, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.5, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 2, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 2, 2024-04-23T21:23:23.715 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.80528640747070312, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.81000000238418579, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 2, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": ".mgr", 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:09.608446+0000", 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 1, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool", 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:23:23.716 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 1, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 1, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 1, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 1, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 1, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:23:23.717 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "17", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 0, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 0, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:23:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:23:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:23:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: "options": { 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_max": 32, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_min": 1 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": {} 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 7.690000057220459, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 7.690000057220459, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.12999999523162842, 2024-04-23T21:23:23.721 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.80528640747070312, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.81000000238418579, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_name": "pool_client.0", 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "create_time": "2024-04-23T21:21:16.010288+0000", 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "flags": 8193, 2024-04-23T21:23:23.722 INFO:teuthology.orchestra.run.smithi111.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "type": 1, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 2, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "min_size": 1, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_rule": 0, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_count": 0, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_target": 0, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_barrier": 0, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "object_hash": 2, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_autoscale_mode": "off", 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num": 32, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num": 32, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_placement_num_target": 32, 2024-04-23T21:23:23.723 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_target": 32, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_num_pending": 32, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_pg_merge_meta": { 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "source_pgid": "0.0", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "ready_epoch": 0, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_started": 0, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_epoch_clean": 0, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "source_version": "0'0", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "target_version": "0'0" 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_change": "139", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend": "0", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "last_force_op_resend_preluminous": "0", 2024-04-23T21:23:23.724 INFO:teuthology.orchestra.run.smithi111.stdout: "auid": 0, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_mode": "selfmanaged", 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_seq": 51, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "snap_epoch": 139, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "pool_snaps": [], 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps": "[]", 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_bytes": 0, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "quota_max_objects": 0, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "tiers": [], 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "tier_of": -1, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "read_tier": -1, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "write_tier": -1, 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_mode": "none", 2024-04-23T21:23:23.725 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_bytes": 0, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "target_max_objects": 0, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_flush_age": 0, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "cache_min_evict_age": 0, 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profile": "", 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_params": { 2024-04-23T21:23:23.726 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "none" 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_period": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_count": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "use_gmt_hitset": true, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "min_read_recency_for_promote": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "min_write_recency_for_promote": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_grade_decay_rate": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "hit_set_search_last_n": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "grade_table": [], 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "stripe_width": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "expected_num_objects": 0, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "fast_read": false, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "options": {}, 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "application_metadata": { 2024-04-23T21:23:23.727 INFO:teuthology.orchestra.run.smithi111.stdout: "rbd": {} 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "read_balance": { 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "score_type": "Fair distribution", 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "score_acting": 3.7599999904632568, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "score_stable": 3.7599999904632568, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "optimal_score": 0.5, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_acting": 1.8799999952316284, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "raw_score_stable": 1.8799999952316284, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity_weighted": 0.80528640747070312, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity": 0.81000000238418579, 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "average_primary_affinity_weighted": 0.81000000238418579 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.728 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "30c47d3d-583d-4987-9b86-247b80cb80ae", 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.04681396484375, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 130, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.729 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6808", 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6809", 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6810", 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6811", 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.731 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6814", 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6815", 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.732 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6812", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6813", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1367522096 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6809/1367522096", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6811/1367522096", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6815/1367522096", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6813/1367522096", 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.733 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d7c695c5-c361-4c22-b4e7-bd16572f0550", 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 13, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 17, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 45, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 137, 2024-04-23T21:23:23.734 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 18, 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6816", 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6817", 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6818", 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6819", 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.736 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6822", 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6823", 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6820", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6821", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 723341542 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6817/723341542", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6819/723341542", 2024-04-23T21:23:23.738 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6823/723341542", 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6821/723341542", 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "46e4a0e9-b403-4a09-92ca-57245c3fec30", 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.739 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 131, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6800", 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6801", 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6802", 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.741 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6803", 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6806", 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.742 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6807", 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6804", 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.743 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6805", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 698077721 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6801/698077721", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6803/698077721", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6807/698077721", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6805/698077721", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.744 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "c18de0c4-3fd5-46a0-85a6-f813c8202ea8", 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 47, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.745 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6824", 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6825", 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.746 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6826", 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6827", 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.747 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6830", 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6831", 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6828", 2024-04-23T21:23:23.748 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.111:6829", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 4030519478 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.111:6825/4030519478", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.111:6827/4030519478", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.111:6831/4030519478", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.111:6829/4030519478", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.749 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "d0562ad9-c051-475a-a40b-248d222c8921", 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 138, 2024-04-23T21:23:23.750 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6816", 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6817", 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.751 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6818", 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6819", 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.752 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6822", 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6824", 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.753 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6820", 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6821", 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1513005997 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.754 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6817/1513005997", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6819/1513005997", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6824/1513005997", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6821/1513005997", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "344f8beb-b979-4248-92df-c34cd560be6a", 2024-04-23T21:23:23.755 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 0, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 0.395477294921875, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 88, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 105, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6800", 2024-04-23T21:23:23.756 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6801", 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6802", 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.757 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6803", 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6806", 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.758 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6807", 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6804", 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.759 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6805", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2687705294 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6801/2687705294", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6803/2687705294", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6807/2687705294", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6805/2687705294", 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "exists" 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.760 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "62434311-7dc6-4a3e-bd07-1e8915b37709", 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 1, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 1, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 137, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.761 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6823", 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6825", 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.762 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6826", 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6827", 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6830", 2024-04-23T21:23:23.763 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6831", 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6828", 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.764 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6829", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 2824610433 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6825/2824610433", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6827/2824610433", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6831/2824610433", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6829/2824610433", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.765 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "uuid": "db260d92-d3d7-464e-ab34-0bedd2991992", 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "up": 1, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "in": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "weight": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_affinity": 1, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_begin": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "last_clean_end": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "up_from": 13, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "up_thru": 136, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "down_at": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "lost_at": 0, 2024-04-23T21:23:23.766 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addrs": { 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6808", 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6809", 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addrs": { 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.767 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6810", 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6811", 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addrs": { 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.768 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6814", 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6815", 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addrs": { 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "addrvec": [ 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v2", 2024-04-23T21:23:23.769 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6812", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "type": "v1", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "addr": "172.21.15.192:6813", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "nonce": 1007335403 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "public_addr": "172.21.15.192:6809/1007335403", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster_addr": "172.21.15.192:6811/1007335403", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_back_addr": "172.21.15.192:6815/1007335403", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "heartbeat_front_addr": "172.21.15.192:6813/1007335403", 2024-04-23T21:23:23.770 INFO:teuthology.orchestra.run.smithi111.stdout: "state": [ 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "exists", 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "up" 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "osd_xinfo": [ 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 0, 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 65536, 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.914033+0000", 2024-04-23T21:23:23.771 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 1, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.903496+0000", 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 18 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 2, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.772 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.934930+0000", 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 3, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.773 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 65536, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:05.001485+0000", 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 4, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.979253+0000", 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.774 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 5, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.839146+0000", 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 105 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 6, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.775 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 0, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.963622+0000", 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": 7, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "down_stamp": "0.000000", 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_probability": 0, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "laggy_interval": 0, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "features": 4540701547738038271, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "old_weight": 65536, 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "last_purged_snaps_scrub": "2024-04-23T21:21:04.946000+0000", 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: "dead_epoch": 0 2024-04-23T21:23:23.776 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap": [], 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_items": [], 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_upmap_primaries": [], 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pg_temp": [ 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.1", 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: 2, 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: 0 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.4", 2024-04-23T21:23:23.777 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: 6, 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "1.7", 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: 6, 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "2.0", 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.778 INFO:teuthology.orchestra.run.smithi111.stdout: 1, 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.1", 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: 4, 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: 0 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.7", 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: 4, 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.779 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.d", 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: 6, 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.10", 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: 6, 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: 0 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.780 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.16", 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: 1, 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: 7 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.17", 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: 6, 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: 0 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.781 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "pgid": "3.1f", 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "osds": [ 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: 2, 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: 0 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "primary_temp": [], 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "blocklist": {}, 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "range_blocklist": {}, 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "erasure_code_profiles": { 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "default": { 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "crush-failure-domain": "osd", 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "k": "2", 2024-04-23T21:23:23.782 INFO:teuthology.orchestra.run.smithi111.stdout: "m": "1", 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "plugin": "jerasure", 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "technique": "reed_sol_van" 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "removed_snaps_queue": [ 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 16, 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 25 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.783 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "new_removed_snaps": [ 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "pool": 3, 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "snaps": [ 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "begin": 16, 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "length": 1 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: ], 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "new_purged_snaps": [], 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "crush_node_flags": {}, 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "device_class_flags": {}, 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode": { 2024-04-23T21:23:23.784 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_enabled": false, 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_bucket_count": 0, 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout: "degraded_stretch_mode": 0, 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout: "recovering_stretch_mode": 0, 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout: "stretch_mode_bucket": 0 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:23.785 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.786 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.786 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:23.786 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:23.786 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.788 INFO:tasks.thrashosds.thrasher:j is {'epoch': 139, 'fsid': '3556a001-ee44-4b6e-8d50-1bc45ce74758', 'created': '2024-04-23T21:20:57.391123+0000', 'modified': '2024-04-23T21:23:23.189206+0000', 'last_up_change': '2024-04-23T21:22:49.185080+0000', 'last_in_change': '2024-04-23T21:23:18.371965+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-23T21:21:08.380744+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8052864074707031, 'average_primary_affinity': 0.8100000023841858, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-23T21:21:09.608446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.690000057220459, 'score_stable': 7.690000057220459, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 1, 'raw_score_stable': 1, 'primary_affinity_weighted': 0.8052864074707031, 'average_primary_affinity': 0.8100000023841858, 'average_primary_affinity_weighted': 0.8100000023841858}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-04-23T21:21:16.010288+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 51, 'snap_epoch': 139, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.759999990463257, 'score_stable': 3.759999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8052864074707031, 'average_primary_affinity': 0.8100000023841858, 'average_primary_affinity_weighted': 0.8100000023841858}}], 'osds': [{'osd': 0, 'uuid': '30c47d3d-583d-4987-9b86-247b80cb80ae', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0.04681396484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 130, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6808', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6809', 'nonce': 1367522096}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6810', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6811', 'nonce': 1367522096}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6814', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6815', 'nonce': 1367522096}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6812', 'nonce': 1367522096}, {'type': 'v1', 'addr': '172.21.15.111:6813', 'nonce': 1367522096}]}, 'public_addr': '172.21.15.111:6809/1367522096', 'cluster_addr': '172.21.15.111:6811/1367522096', 'heartbeat_back_addr': '172.21.15.111:6815/1367522096', 'heartbeat_front_addr': '172.21.15.111:6813/1367522096', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'd7c695c5-c361-4c22-b4e7-bd16572f0550', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 17, 'up_from': 45, 'up_thru': 137, 'down_at': 18, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6816', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6817', 'nonce': 723341542}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6818', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6819', 'nonce': 723341542}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6822', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6823', 'nonce': 723341542}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6820', 'nonce': 723341542}, {'type': 'v1', 'addr': '172.21.15.111:6821', 'nonce': 723341542}]}, 'public_addr': '172.21.15.111:6817/723341542', 'cluster_addr': '172.21.15.111:6819/723341542', 'heartbeat_back_addr': '172.21.15.111:6823/723341542', 'heartbeat_front_addr': '172.21.15.111:6821/723341542', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '46e4a0e9-b403-4a09-92ca-57245c3fec30', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6800', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6801', 'nonce': 698077721}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6802', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6803', 'nonce': 698077721}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6806', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6807', 'nonce': 698077721}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6804', 'nonce': 698077721}, {'type': 'v1', 'addr': '172.21.15.111:6805', 'nonce': 698077721}]}, 'public_addr': '172.21.15.111:6801/698077721', 'cluster_addr': '172.21.15.111:6803/698077721', 'heartbeat_back_addr': '172.21.15.111:6807/698077721', 'heartbeat_front_addr': '172.21.15.111:6805/698077721', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'c18de0c4-3fd5-46a0-85a6-f813c8202ea8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 47, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6824', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6825', 'nonce': 4030519478}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6826', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6827', 'nonce': 4030519478}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6830', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6831', 'nonce': 4030519478}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.111:6828', 'nonce': 4030519478}, {'type': 'v1', 'addr': '172.21.15.111:6829', 'nonce': 4030519478}]}, 'public_addr': '172.21.15.111:6825/4030519478', 'cluster_addr': '172.21.15.111:6827/4030519478', 'heartbeat_back_addr': '172.21.15.111:6831/4030519478', 'heartbeat_front_addr': '172.21.15.111:6829/4030519478', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'd0562ad9-c051-475a-a40b-248d222c8921', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 138, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6816', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6817', 'nonce': 1513005997}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6818', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6819', 'nonce': 1513005997}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6822', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6824', 'nonce': 1513005997}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6820', 'nonce': 1513005997}, {'type': 'v1', 'addr': '172.21.15.192:6821', 'nonce': 1513005997}]}, 'public_addr': '172.21.15.192:6817/1513005997', 'cluster_addr': '172.21.15.192:6819/1513005997', 'heartbeat_back_addr': '172.21.15.192:6824/1513005997', 'heartbeat_front_addr': '172.21.15.192:6821/1513005997', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '344f8beb-b979-4248-92df-c34cd560be6a', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 0.395477294921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 88, 'down_at': 105, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6800', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6801', 'nonce': 2687705294}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6802', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6803', 'nonce': 2687705294}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6806', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6807', 'nonce': 2687705294}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6804', 'nonce': 2687705294}, {'type': 'v1', 'addr': '172.21.15.192:6805', 'nonce': 2687705294}]}, 'public_addr': '172.21.15.192:6801/2687705294', 'cluster_addr': '172.21.15.192:6803/2687705294', 'heartbeat_back_addr': '172.21.15.192:6807/2687705294', 'heartbeat_front_addr': '172.21.15.192:6805/2687705294', 'state': ['exists']}, {'osd': 6, 'uuid': '62434311-7dc6-4a3e-bd07-1e8915b37709', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 137, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6823', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6825', 'nonce': 2824610433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6826', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6827', 'nonce': 2824610433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6830', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6831', 'nonce': 2824610433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6828', 'nonce': 2824610433}, {'type': 'v1', 'addr': '172.21.15.192:6829', 'nonce': 2824610433}]}, 'public_addr': '172.21.15.192:6825/2824610433', 'cluster_addr': '172.21.15.192:6827/2824610433', 'heartbeat_back_addr': '172.21.15.192:6831/2824610433', 'heartbeat_front_addr': '172.21.15.192:6829/2824610433', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'db260d92-d3d7-464e-ab34-0bedd2991992', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 136, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6808', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6809', 'nonce': 1007335403}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6810', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6811', 'nonce': 1007335403}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6814', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6815', 'nonce': 1007335403}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6812', 'nonce': 1007335403}, {'type': 'v1', 'addr': '172.21.15.192:6813', 'nonce': 1007335403}]}, 'public_addr': '172.21.15.192:6809/1007335403', 'cluster_addr': '172.21.15.192:6811/1007335403', 'heartbeat_back_addr': '172.21.15.192:6815/1007335403', 'heartbeat_front_addr': '172.21.15.192:6813/1007335403', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.914033+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.903496+0000', 'dead_epoch': 18}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.934930+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T21:21:05.001485+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.979253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.839146+0000', 'dead_epoch': 105}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.963622+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-23T21:21:04.946000+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.1', 'osds': [2, 0]}, {'pgid': '1.4', 'osds': [6, 7]}, {'pgid': '1.7', 'osds': [6, 7]}, {'pgid': '2.0', 'osds': [1, 7]}, {'pgid': '3.1', 'osds': [4, 0]}, {'pgid': '3.7', 'osds': [4, 7]}, {'pgid': '3.d', 'osds': [6, 7]}, {'pgid': '3.10', 'osds': [6, 0]}, {'pgid': '3.16', 'osds': [1, 7]}, {'pgid': '3.17', 'osds': [6, 0]}, {'pgid': '3.1f', 'osds': [2, 0]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 16, 'length': 25}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 16, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-23T21:23:23.788 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:23:23.789 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:23.789 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:23.787+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:23.887 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:23.888+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:23.932 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:23.933 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:23.933 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:23.933 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:23.933 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:23.942 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:23.988 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:23.988+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:24.047 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.407918643000002, 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.048 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.049 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:24.050 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:24.051 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.407886536000007, 2024-04-23T21:23:24.052 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.053 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:24.054 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:24.055 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.056 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.407863149999997, 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:24.057 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:24.058 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:24.059 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.060 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.407842842999997, 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.061 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.062 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.063 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.064 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.407823031000007, 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.065 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.066 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:24.067 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:24.068 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406751632999999, 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.069 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.070 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.071 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.072 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406718287999993, 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.073 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.074 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:24.075 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.076 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406686758999996, 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.077 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.078 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:24.079 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:24.080 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.40665937, 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.081 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.082 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.083 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:24.084 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:24.084 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.084 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.086 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406647464000002, 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:24.087 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.088 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.089 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.090 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406629219999999, 2024-04-23T21:23:24.091 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.092 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.093 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.094 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.095 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406619402999993, 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.096 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:24.097 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:24.098 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.099 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406609156000002, 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:24.100 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:24.101 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:24.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.103 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406599505000003, 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.406566006000006, 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.111 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.402940581999999, 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.112 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.402922712999995, 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.116 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.117 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.118 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.402903984000005, 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.120 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:24.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:24.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.402885556000001, 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.124 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:24.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:24.128 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 97.402867650999994, 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:24.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:24.132 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:24.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:24.135 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:23:24.136 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:23:24.137 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-23T21:23:23.617533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":805656340,"num_objects":1025,"num_object_clones":118,"num_object_copies":2050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":147,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":686,"num_read":184777,"num_read_kb":1008016,"num_write":19352,"num_write_kb":589415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":123547689,"num_keys_recovered":256,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16991,"ondisk_log_size":16991,"up":66,"acting":74,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":233392,"kb_used_data":27560,"kb_used_omap":12,"kb_used_meta":205747,"kb_avail":471625808,"statfs":{"total":483183820800,"available":482944827392,"internally_reserved":0,"allocated":28221440,"data_stored":145266898,"data_compressed":7896354,"data_compressed_allocated":9568256,"data_compressed_original":132161945,"omap_allocated":13126,"internal_metadata":210685114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23939023,"num_objects":-26,"num_object_clones":-10,"num_object_copies":-52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-41,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":-8,"num_read":301,"num_read_kb":262,"num_write":46,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":384,"num_bytes_recovered":3467565,"num_keys_recovered":12,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000832"},"pg_stats":[{"pgid":"3.1f","version":"134'489","reported_seq":4672,"reported_epoch":134,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:18.385723+0000","last_change":"2024-04-23T21:23:18.385723+0000","last_active":"2024-04-23T21:23:18.385723+0000","last_peered":"2024-04-23T21:23:18.385723+0000","last_clean":"2024-04-23T21:23:18.385723+0000","last_became_active":"2024-04-23T21:23:16.367907+0000","last_became_peered":"2024-04-23T21:23:16.367907+0000","last_unstale":"2024-04-23T21:23:18.385723+0000","last_undegraded":"2024-04-23T21:23:18.385723+0000","last_fullsized":"2024-04-23T21:23:18.385723+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:51:03.026436+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0058607779999999996,"stat_sum":{"num_bytes":30356039,"num_objects":29,"num_object_clones":1,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":20,"num_read":6769,"num_read_kb":17445,"num_write":517,"num_write_kb":33867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.1e","version":"134'437","reported_seq":6933,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.392403+0000","last_change":"2024-04-23T21:23:18.392665+0000","last_active":"2024-04-23T21:23:18.392403+0000","last_peered":"2024-04-23T21:23:18.392403+0000","last_clean":"2024-04-23T21:23:18.392403+0000","last_became_active":"2024-04-23T21:22:33.186559+0000","last_became_peered":"2024-04-23T21:22:33.186559+0000","last_unstale":"2024-04-23T21:23:18.392403+0000","last_undegraded":"2024-04-23T21:23:18.392403+0000","last_fullsized":"2024-04-23T21:23:18.392403+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:41:06.724722+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0032244800000000001,"stat_sum":{"num_bytes":24352459,"num_objects":35,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":22,"num_read":6453,"num_read_kb":47260,"num_write":468,"num_write_kb":24906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.1d","version":"126'304","reported_seq":8362,"reported_epoch":134,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:23:18.386517+0000","last_change":"2024-04-23T21:22:50.200976+0000","last_active":"2024-04-23T21:23:18.386517+0000","last_peered":"2024-04-23T21:23:18.386517+0000","last_clean":"2024-04-23T21:22:48.222633+0000","last_became_active":"2024-04-23T21:22:50.200976+0000","last_became_peered":"2024-04-23T21:22:50.200976+0000","last_unstale":"2024-04-23T21:23:18.386517+0000","last_undegraded":"2024-04-23T21:22:50.199987+0000","last_fullsized":"2024-04-23T21:22:50.199902+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:30:21.253695+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0033353929999999999,"stat_sum":{"num_bytes":27008995,"num_objects":44,"num_object_clones":7,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":44,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":29,"num_read":8314,"num_read_kb":31845,"num_write":273,"num_write_kb":7752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":44}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"133'451","reported_seq":3306,"reported_epoch":134,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:23:18.386065+0000","last_change":"2024-04-23T21:22:50.200181+0000","last_active":"2024-04-23T21:23:18.386065+0000","last_peered":"2024-04-23T21:23:18.386065+0000","last_clean":"2024-04-23T21:22:48.240944+0000","last_became_active":"2024-04-23T21:22:50.200181+0000","last_became_peered":"2024-04-23T21:22:50.200181+0000","last_unstale":"2024-04-23T21:23:18.386065+0000","last_undegraded":"2024-04-23T21:22:50.199507+0000","last_fullsized":"2024-04-23T21:22:50.199419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:15:00.728055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029319350000000001,"stat_sum":{"num_bytes":61620256,"num_objects":61,"num_object_clones":14,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":61,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":36,"num_read":3147,"num_read_kb":24903,"num_write":537,"num_write_kb":16868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[{"shards":"4","objects":61}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"127'408","reported_seq":10749,"reported_epoch":133,"state":"peering","last_fresh":"2024-04-23T21:23:17.377383+0000","last_change":"2024-04-23T21:23:18.388200+0000","last_active":"2024-04-23T21:23:18.388200+0000","last_peered":"2024-04-23T21:23:17.377383+0000","last_clean":"2024-04-23T21:23:17.377383+0000","last_became_active":"2024-04-23T21:23:16.369213+0000","last_became_peered":"2024-04-23T21:23:16.369213+0000","last_unstale":"2024-04-23T21:23:17.377383+0000","last_undegraded":"2024-04-23T21:23:17.377383+0000","last_fullsized":"2024-04-23T21:23:17.377383+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001677764,"stat_sum":{"num_bytes":45195485,"num_objects":39,"num_object_clones":8,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":21,"num_read":11590,"num_read_kb":51962,"num_write":382,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":6389369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,7],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"3.1a","version":"134'883","reported_seq":6350,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.398008+0000","last_change":"2024-04-23T21:23:18.389600+0000","last_active":"2024-04-23T21:23:18.398008+0000","last_peered":"2024-04-23T21:23:18.398008+0000","last_clean":"2024-04-23T21:23:18.398008+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:23:18.398008+0000","last_undegraded":"2024-04-23T21:23:18.398008+0000","last_fullsized":"2024-04-23T21:23:18.398008+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":883,"log_dups_size":0,"ondisk_log_size":883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:15:58.507984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003192458,"stat_sum":{"num_bytes":2201062,"num_objects":31,"num_object_clones":2,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":22,"num_read":9994,"num_read_kb":52488,"num_write":1069,"num_write_kb":37931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.19","version":"133'848","reported_seq":3075,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.190749+0000","last_change":"2024-04-23T21:23:21.392154+0000","last_active":"2024-04-23T21:23:22.190749+0000","last_peered":"2024-04-23T21:23:22.190749+0000","last_clean":"2024-04-23T21:23:22.190749+0000","last_became_active":"2024-04-23T21:22:34.185924+0000","last_became_peered":"2024-04-23T21:22:34.185924+0000","last_unstale":"2024-04-23T21:23:22.190749+0000","last_undegraded":"2024-04-23T21:23:22.190749+0000","last_fullsized":"2024-04-23T21:23:22.190749+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:10:55.752265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00152406,"stat_sum":{"num_bytes":16734498,"num_objects":33,"num_object_clones":3,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":25,"num_read":5388,"num_read_kb":14837,"num_write":1126,"num_write_kb":9876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.18","version":"138'720","reported_seq":6086,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.276195+0000","last_change":"2024-04-23T21:23:21.392018+0000","last_active":"2024-04-23T21:23:22.276195+0000","last_peered":"2024-04-23T21:23:22.276195+0000","last_clean":"2024-04-23T21:23:22.276195+0000","last_became_active":"2024-04-23T21:23:16.368322+0000","last_became_peered":"2024-04-23T21:23:16.368322+0000","last_unstale":"2024-04-23T21:23:22.276195+0000","last_undegraded":"2024-04-23T21:23:22.276195+0000","last_fullsized":"2024-04-23T21:23:22.276195+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:37:48.611108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014732320000000001,"stat_sum":{"num_bytes":15959367,"num_objects":32,"num_object_clones":3,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":24,"num_read":8069,"num_read_kb":25158,"num_write":862,"num_write_kb":7579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.17","version":"133'583","reported_seq":3148,"reported_epoch":134,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:18.391127+0000","last_change":"2024-04-23T21:23:18.391127+0000","last_active":"2024-04-23T21:23:18.391127+0000","last_peered":"2024-04-23T21:23:18.391127+0000","last_clean":"2024-04-23T21:23:18.391127+0000","last_became_active":"2024-04-23T21:23:16.370802+0000","last_became_peered":"2024-04-23T21:23:16.370802+0000","last_unstale":"2024-04-23T21:23:18.391127+0000","last_undegraded":"2024-04-23T21:23:18.391127+0000","last_fullsized":"2024-04-23T21:23:18.391127+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":583,"log_dups_size":0,"ondisk_log_size":583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:46:36.664044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015568209999999999,"stat_sum":{"num_bytes":6874590,"num_objects":36,"num_object_clones":1,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":29,"num_read":4053,"num_read_kb":30935,"num_write":598,"num_write_kb":28221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":36}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.16","version":"138'458","reported_seq":5450,"reported_epoch":138,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:22.243743+0000","last_change":"2024-04-23T21:23:22.243743+0000","last_active":"2024-04-23T21:23:22.243743+0000","last_peered":"2024-04-23T21:23:22.243743+0000","last_clean":"2024-04-23T21:23:22.243743+0000","last_became_active":"2024-04-23T21:23:22.196345+0000","last_became_peered":"2024-04-23T21:23:22.196345+0000","last_unstale":"2024-04-23T21:23:22.243743+0000","last_undegraded":"2024-04-23T21:23:22.243743+0000","last_fullsized":"2024-04-23T21:23:22.243743+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:24:59.524055+0000","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":0.046268997999999999,"stat_sum":{"num_bytes":5295391,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":469,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":24}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.15","version":"131'367","reported_seq":4663,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.386169+0000","last_change":"2024-04-23T21:23:18.627639+0000","last_active":"2024-04-23T21:23:18.386169+0000","last_peered":"2024-04-23T21:23:15.367276+0000","last_clean":"2024-04-23T21:23:15.367276+0000","last_became_active":"2024-04-23T21:22:33.191679+0000","last_became_peered":"2024-04-23T21:22:33.191679+0000","last_unstale":"2024-04-23T21:23:18.386169+0000","last_undegraded":"2024-04-23T21:23:18.386169+0000","last_fullsized":"2024-04-23T21:23:18.386169+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.002331577,"stat_sum":{"num_bytes":30387344,"num_objects":40,"num_object_clones":6,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":23,"num_read":4322,"num_read_kb":26762,"num_write":346,"num_write_kb":9614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":5579029,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"3.14","version":"127'649","reported_seq":5951,"reported_epoch":133,"state":"peering","last_fresh":"2024-04-23T21:23:17.376750+0000","last_change":"2024-04-23T21:23:18.387260+0000","last_active":"2024-04-23T21:23:18.387260+0000","last_peered":"2024-04-23T21:23:17.376750+0000","last_clean":"2024-04-23T21:23:17.376750+0000","last_became_active":"2024-04-23T21:21:17.780684+0000","last_became_peered":"2024-04-23T21:21:17.780684+0000","last_unstale":"2024-04-23T21:23:17.376750+0000","last_undegraded":"2024-04-23T21:23:17.376750+0000","last_fullsized":"2024-04-23T21:23:17.376750+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017560480000000001,"stat_sum":{"num_bytes":53681987,"num_objects":43,"num_object_clones":7,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":7673,"num_read_kb":52269,"num_write":784,"num_write_kb":17081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"1.7","version":"0'0","reported_seq":253,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.388896+0000","last_change":"2024-04-23T21:23:18.625990+0000","last_active":"2024-04-23T21:23:18.388896+0000","last_peered":"2024-04-23T21:23:17.369796+0000","last_clean":"2024-04-23T21:23:17.369796+0000","last_became_active":"2024-04-23T21:22:30.358837+0000","last_became_peered":"2024-04-23T21:22:30.358837+0000","last_unstale":"2024-04-23T21:23:18.388896+0000","last_undegraded":"2024-04-23T21:23:18.388896+0000","last_fullsized":"2024-04-23T21:23:18.388896+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"133'450","reported_seq":2821,"reported_epoch":134,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:23:18.380064+0000","last_change":"2024-04-23T21:22:50.201625+0000","last_active":"2024-04-23T21:23:18.380064+0000","last_peered":"2024-04-23T21:23:18.380064+0000","last_clean":"2024-04-23T21:22:48.222469+0000","last_became_active":"2024-04-23T21:22:50.201625+0000","last_became_peered":"2024-04-23T21:22:50.201625+0000","last_unstale":"2024-04-23T21:23:18.380064+0000","last_undegraded":"2024-04-23T21:22:50.200678+0000","last_fullsized":"2024-04-23T21:22:50.200572+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:38:43.389634+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0049865739999999997,"stat_sum":{"num_bytes":49839986,"num_objects":42,"num_object_clones":11,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":23,"num_read":2650,"num_read_kb":19048,"num_write":515,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":10723491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":42}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":134,"state":"peering","last_fresh":"2024-04-23T21:23:13.340406+0000","last_change":"2024-04-23T21:23:12.340526+0000","last_active":"2024-04-23T21:23:12.340526+0000","last_peered":"2024-04-23T21:23:11.339954+0000","last_clean":"2024-04-23T21:23:11.339954+0000","last_became_active":"2024-04-23T21:22:04.355277+0000","last_became_peered":"2024-04-23T21:22:04.355277+0000","last_unstale":"2024-04-23T21:23:13.340406+0000","last_undegraded":"2024-04-23T21:23:13.340406+0000","last_fullsized":"2024-04-23T21:23:13.340406+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"131'434","reported_seq":4386,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.191351+0000","last_change":"2024-04-23T21:23:21.392811+0000","last_active":"2024-04-23T21:23:22.191351+0000","last_peered":"2024-04-23T21:23:22.191351+0000","last_clean":"2024-04-23T21:23:22.191351+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:23:22.191351+0000","last_undegraded":"2024-04-23T21:23:22.191351+0000","last_fullsized":"2024-04-23T21:23:22.191351+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:14:45.338574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020995440000000001,"stat_sum":{"num_bytes":35838477,"num_objects":23,"num_object_clones":3,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":13,"num_read":5175,"num_read_kb":34978,"num_write":360,"num_write_kb":17883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"1.5","version":"0'0","reported_seq":249,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.386790+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:23:18.386790+0000","last_peered":"2024-04-23T21:23:18.386790+0000","last_clean":"2024-04-23T21:23:18.386790+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:23:18.386790+0000","last_undegraded":"2024-04-23T21:23:18.386790+0000","last_fullsized":"2024-04-23T21:23:18.386790+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"130'414","reported_seq":3782,"reported_epoch":133,"state":"remapped+peering","last_fresh":"2024-04-23T21:23:17.372944+0000","last_change":"2024-04-23T21:23:18.384342+0000","last_active":"2024-04-23T21:23:18.384342+0000","last_peered":"2024-04-23T21:23:17.372944+0000","last_clean":"2024-04-23T21:23:17.372944+0000","last_became_active":"2024-04-23T21:23:14.347329+0000","last_became_peered":"2024-04-23T21:23:14.347329+0000","last_unstale":"2024-04-23T21:23:17.372944+0000","last_undegraded":"2024-04-23T21:23:17.372944+0000","last_fullsized":"2024-04-23T21:23:17.372944+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":414,"log_dups_size":0,"ondisk_log_size":414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001832836,"stat_sum":{"num_bytes":27272944,"num_objects":37,"num_object_clones":3,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":27,"num_read":3779,"num_read_kb":24522,"num_write":468,"num_write_kb":11833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":9761436,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":4,"acting_primary":7,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"1.4","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.388279+0000","last_change":"2024-04-23T21:23:18.625848+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T21:23:18.388279+0000","last_undegraded":"2024-04-23T21:23:18.388279+0000","last_fullsized":"2024-04-23T21:23:18.388279+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"138'405","reported_seq":4627,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.191382+0000","last_change":"2024-04-23T21:23:21.393217+0000","last_active":"2024-04-23T21:23:22.191382+0000","last_peered":"2024-04-23T21:23:22.191382+0000","last_clean":"2024-04-23T21:23:22.191382+0000","last_became_active":"2024-04-23T21:23:16.368081+0000","last_became_peered":"2024-04-23T21:23:16.368081+0000","last_unstale":"2024-04-23T21:23:22.191382+0000","last_undegraded":"2024-04-23T21:23:22.191382+0000","last_fullsized":"2024-04-23T21:23:22.191382+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":405,"log_dups_size":0,"ondisk_log_size":405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:35:27.533128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018831900000000001,"stat_sum":{"num_bytes":2157131,"num_objects":33,"num_object_clones":3,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":28,"num_read":5969,"num_read_kb":33589,"num_write":358,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.1","version":"133'409","reported_seq":8619,"reported_epoch":134,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:18.389797+0000","last_change":"2024-04-23T21:23:18.390252+0000","last_active":"2024-04-23T21:23:18.389797+0000","last_peered":"2024-04-23T21:23:18.389797+0000","last_clean":"2024-04-23T21:23:18.389797+0000","last_became_active":"2024-04-23T21:23:14.339072+0000","last_became_peered":"2024-04-23T21:23:14.339072+0000","last_unstale":"2024-04-23T21:23:18.389797+0000","last_undegraded":"2024-04-23T21:23:18.389797+0000","last_fullsized":"2024-04-23T21:23:18.389797+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:22:34.281850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036888630000000001,"stat_sum":{"num_bytes":52056668,"num_objects":30,"num_object_clones":4,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":30,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":20,"num_read":8322,"num_read_kb":76065,"num_write":489,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":30}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"2.0","version":"16'117","reported_seq":173,"reported_epoch":138,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:22.191030+0000","last_change":"2024-04-23T21:23:20.394436+0000","last_active":"2024-04-23T21:23:22.191030+0000","last_peered":"2024-04-23T21:23:22.191030+0000","last_clean":"2024-04-23T21:23:22.191030+0000","last_became_active":"2024-04-23T21:23:20.393930+0000","last_became_peered":"2024-04-23T21:23:20.393930+0000","last_unstale":"2024-04-23T21:23:22.191030+0000","last_undegraded":"2024-04-23T21:23:22.191030+0000","last_fullsized":"2024-04-23T21:23:22.191030+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:36:15.870370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":204,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.191298+0000","last_change":"2024-04-23T21:21:49.937409+0000","last_active":"2024-04-23T21:23:22.191298+0000","last_peered":"2024-04-23T21:23:22.191298+0000","last_clean":"2024-04-23T21:23:22.191298+0000","last_became_active":"2024-04-23T21:21:49.937122+0000","last_became_peered":"2024-04-23T21:21:49.937122+0000","last_unstale":"2024-04-23T21:23:22.191298+0000","last_undegraded":"2024-04-23T21:23:22.191298+0000","last_fullsized":"2024-04-23T21:23:22.191298+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:19:40.277550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"136'329","reported_seq":1226,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.190834+0000","last_change":"2024-04-23T21:23:21.392721+0000","last_active":"2024-04-23T21:23:22.190834+0000","last_peered":"2024-04-23T21:23:22.190834+0000","last_clean":"2024-04-23T21:23:22.190834+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:23:22.190834+0000","last_undegraded":"2024-04-23T21:23:22.190834+0000","last_fullsized":"2024-04-23T21:23:22.190834+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:32:39.171641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018577310000000001,"stat_sum":{"num_bytes":7093046,"num_objects":28,"num_object_clones":1,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":21,"num_read":3380,"num_read_kb":9704,"num_write":348,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"1.2","version":"0'0","reported_seq":238,"reported_epoch":134,"state":"peering","last_fresh":"2024-04-23T21:23:18.386133+0000","last_change":"2024-04-23T21:23:12.341260+0000","last_active":"2024-04-23T21:23:12.341260+0000","last_peered":"2024-04-23T21:23:11.340451+0000","last_clean":"2024-04-23T21:23:11.340451+0000","last_became_active":"2024-04-23T21:21:09.823433+0000","last_became_peered":"2024-04-23T21:21:09.823433+0000","last_unstale":"2024-04-23T21:23:18.386133+0000","last_undegraded":"2024-04-23T21:23:18.386133+0000","last_fullsized":"2024-04-23T21:23:18.386133+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"134'880","reported_seq":8959,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.392485+0000","last_change":"2024-04-23T21:23:18.392791+0000","last_active":"2024-04-23T21:23:18.392485+0000","last_peered":"2024-04-23T21:23:18.392485+0000","last_clean":"2024-04-23T21:23:18.392485+0000","last_became_active":"2024-04-23T21:23:16.370077+0000","last_became_peered":"2024-04-23T21:23:16.370077+0000","last_unstale":"2024-04-23T21:23:18.392485+0000","last_undegraded":"2024-04-23T21:23:18.392485+0000","last_fullsized":"2024-04-23T21:23:18.392485+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:18:37.509660+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0050947329999999997,"stat_sum":{"num_bytes":28890545,"num_objects":31,"num_object_clones":3,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":20,"num_read":10601,"num_read_kb":58061,"num_write":1091,"num_write_kb":33498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":134,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:18.379705+0000","last_change":"2024-04-23T21:23:14.351621+0000","last_active":"2024-04-23T21:23:18.379705+0000","last_peered":"2024-04-23T21:23:18.379705+0000","last_clean":"2024-04-23T21:23:18.379705+0000","last_became_active":"2024-04-23T21:23:14.351251+0000","last_became_peered":"2024-04-23T21:23:14.351251+0000","last_unstale":"2024-04-23T21:23:18.379705+0000","last_undegraded":"2024-04-23T21:23:18.379705+0000","last_fullsized":"2024-04-23T21:23:18.379705+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:34.207842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"132'609","reported_seq":5278,"reported_epoch":134,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:23:18.389836+0000","last_change":"2024-04-23T21:22:50.200496+0000","last_active":"2024-04-23T21:23:18.389836+0000","last_peered":"2024-04-23T21:23:18.389836+0000","last_clean":"2024-04-23T21:22:46.469672+0000","last_became_active":"2024-04-23T21:22:50.200496+0000","last_became_peered":"2024-04-23T21:22:50.200496+0000","last_unstale":"2024-04-23T21:23:18.389836+0000","last_undegraded":"2024-04-23T21:22:50.198932+0000","last_fullsized":"2024-04-23T21:22:50.198824+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:51:23.968858+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0064004500000000002,"stat_sum":{"num_bytes":32112583,"num_objects":33,"num_object_clones":5,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":20,"num_read":7835,"num_read_kb":15067,"num_write":747,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":33}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":238,"reported_epoch":134,"state":"peering","last_fresh":"2024-04-23T21:23:18.388292+0000","last_change":"2024-04-23T21:23:12.340669+0000","last_active":"2024-04-23T21:23:12.340669+0000","last_peered":"2024-04-23T21:23:11.341152+0000","last_clean":"2024-04-23T21:23:11.341152+0000","last_became_active":"2024-04-23T21:21:09.823375+0000","last_became_peered":"2024-04-23T21:21:09.823375+0000","last_unstale":"2024-04-23T21:23:18.388292+0000","last_undegraded":"2024-04-23T21:23:18.388292+0000","last_fullsized":"2024-04-23T21:23:18.388292+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"131'287","reported_seq":3002,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.198208+0000","last_change":"2024-04-23T21:23:22.198585+0000","last_active":"2024-04-23T21:23:22.198208+0000","last_peered":"2024-04-23T21:23:22.198208+0000","last_clean":"2024-04-23T21:23:22.198208+0000","last_became_active":"2024-04-23T21:23:22.196595+0000","last_became_peered":"2024-04-23T21:23:22.196595+0000","last_unstale":"2024-04-23T21:23:22.198208+0000","last_undegraded":"2024-04-23T21:23:22.198208+0000","last_fullsized":"2024-04-23T21:23:22.198208+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":287,"log_dups_size":0,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:16:50.829386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00117362,"stat_sum":{"num_bytes":5532842,"num_objects":27,"num_object_clones":2,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":20,"num_read":2171,"num_read_kb":15423,"num_write":267,"num_write_kb":12006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.9","version":"134'357","reported_seq":7515,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.392594+0000","last_change":"2024-04-23T21:23:18.392854+0000","last_active":"2024-04-23T21:23:18.392594+0000","last_peered":"2024-04-23T21:23:18.392594+0000","last_clean":"2024-04-23T21:23:18.392594+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:23:18.392594+0000","last_undegraded":"2024-04-23T21:23:18.392594+0000","last_fullsized":"2024-04-23T21:23:18.392594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:27:07.097443+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059290749999999998,"stat_sum":{"num_bytes":24133199,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":19,"num_read":7064,"num_read_kb":46747,"num_write":356,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.a","version":"133'538","reported_seq":3839,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.389803+0000","last_change":"2024-04-23T21:23:18.390586+0000","last_active":"2024-04-23T21:23:18.389803+0000","last_peered":"2024-04-23T21:23:18.389803+0000","last_clean":"2024-04-23T21:23:18.389803+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:23:18.389803+0000","last_undegraded":"2024-04-23T21:23:18.389803+0000","last_fullsized":"2024-04-23T21:23:18.389803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:46:12.925604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011500569999999999,"stat_sum":{"num_bytes":4194357,"num_objects":30,"num_object_clones":1,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":21,"num_read":5708,"num_read_kb":13040,"num_write":512,"num_write_kb":14981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.b","version":"132'459","reported_seq":2777,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-04-23T21:23:18.386463+0000","last_change":"2024-04-23T21:23:18.390428+0000","last_active":"2024-04-23T21:23:18.386463+0000","last_peered":"2024-04-23T21:23:18.386463+0000","last_clean":"2024-04-23T21:23:18.386463+0000","last_became_active":"2024-04-23T21:23:16.372080+0000","last_became_peered":"2024-04-23T21:23:16.372080+0000","last_unstale":"2024-04-23T21:23:18.386463+0000","last_undegraded":"2024-04-23T21:23:18.386463+0000","last_fullsized":"2024-04-23T21:23:18.386463+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:36:12.386492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043800999999999996,"stat_sum":{"num_bytes":18519192,"num_objects":31,"num_object_clones":2,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":18,"num_read":2652,"num_read_kb":25854,"num_write":472,"num_write_kb":14312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.c","version":"133'408","reported_seq":1919,"reported_epoch":134,"state":"active+undersized+degraded","last_fresh":"2024-04-23T21:23:18.389207+0000","last_change":"2024-04-23T21:22:50.200455+0000","last_active":"2024-04-23T21:23:18.389207+0000","last_peered":"2024-04-23T21:23:18.389207+0000","last_clean":"2024-04-23T21:22:47.203136+0000","last_became_active":"2024-04-23T21:22:50.200455+0000","last_became_peered":"2024-04-23T21:22:50.200455+0000","last_unstale":"2024-04-23T21:23:18.389207+0000","last_undegraded":"2024-04-23T21:22:50.198941+0000","last_fullsized":"2024-04-23T21:22:50.198846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:20:27.792033+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.018964399,"stat_sum":{"num_bytes":60115994,"num_objects":46,"num_object_clones":12,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":46,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":29,"num_read":1674,"num_read_kb":18858,"num_write":370,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[{"shards":"6","objects":46}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.388548+0000","last_change":"2024-04-23T21:23:18.625868+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-23T21:23:18.388548+0000","last_undegraded":"2024-04-23T21:23:18.388548+0000","last_fullsized":"2024-04-23T21:23:18.388548+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"130'610","reported_seq":2392,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.386262+0000","last_change":"2024-04-23T21:23:18.627660+0000","last_active":"2024-04-23T21:23:18.386262+0000","last_peered":"2024-04-23T21:23:14.338695+0000","last_clean":"2024-04-23T21:23:14.338695+0000","last_became_active":"2024-04-23T21:21:17.777318+0000","last_became_peered":"2024-04-23T21:21:17.777318+0000","last_unstale":"2024-04-23T21:23:18.386262+0000","last_undegraded":"2024-04-23T21:23:18.386262+0000","last_fullsized":"2024-04-23T21:23:18.386262+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023598099999999999,"stat_sum":{"num_bytes":7728596,"num_objects":22,"num_object_clones":1,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":19,"num_read":3849,"num_read_kb":11542,"num_write":710,"num_write_kb":15964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"3.f","version":"133'1031","reported_seq":10580,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.386467+0000","last_change":"2024-04-23T21:23:18.627670+0000","last_active":"2024-04-23T21:23:18.386467+0000","last_peered":"2024-04-23T21:23:17.448579+0000","last_clean":"2024-04-23T21:23:17.448579+0000","last_became_active":"2024-04-23T21:21:17.779960+0000","last_became_peered":"2024-04-23T21:21:17.779960+0000","last_unstale":"2024-04-23T21:23:18.386467+0000","last_undegraded":"2024-04-23T21:23:18.386467+0000","last_fullsized":"2024-04-23T21:23:18.386467+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1031,"log_dups_size":0,"ondisk_log_size":1031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019204389999999999,"stat_sum":{"num_bytes":37404793,"num_objects":31,"num_object_clones":1,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":17,"num_read":13809,"num_read_kb":59280,"num_write":1358,"num_write_kb":16431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"3.10","version":"133'529","reported_seq":3682,"reported_epoch":134,"state":"active+clean+remapped","last_fresh":"2024-04-23T21:23:18.389697+0000","last_change":"2024-04-23T21:23:18.390196+0000","last_active":"2024-04-23T21:23:18.389697+0000","last_peered":"2024-04-23T21:23:18.389697+0000","last_clean":"2024-04-23T21:23:18.389697+0000","last_became_active":"2024-04-23T21:23:14.338558+0000","last_became_peered":"2024-04-23T21:23:14.338558+0000","last_unstale":"2024-04-23T21:23:18.389697+0000","last_undegraded":"2024-04-23T21:23:18.389697+0000","last_fullsized":"2024-04-23T21:23:18.389697+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:38:31.705659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010078140000000001,"stat_sum":{"num_bytes":52311981,"num_objects":26,"num_object_clones":1,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":18,"num_read":4662,"num_read_kb":33972,"num_write":549,"num_write_kb":20712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":26}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"17"}]},{"pgid":"3.11","version":"131'302","reported_seq":2473,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.385958+0000","last_change":"2024-04-23T21:23:18.627563+0000","last_active":"2024-04-23T21:23:18.385958+0000","last_peered":"2024-04-23T21:23:15.364376+0000","last_clean":"2024-04-23T21:23:15.364376+0000","last_became_active":"2024-04-23T21:21:17.781179+0000","last_became_peered":"2024-04-23T21:21:17.781179+0000","last_unstale":"2024-04-23T21:23:18.385958+0000","last_undegraded":"2024-04-23T21:23:18.385958+0000","last_fullsized":"2024-04-23T21:23:18.385958+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001891724,"stat_sum":{"num_bytes":13197213,"num_objects":17,"num_object_clones":1,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":11,"num_read":1896,"num_read_kb":20602,"num_write":300,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]},{"pgid":"3.12","version":"138'925","reported_seq":3980,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-04-23T21:23:22.302129+0000","last_change":"2024-04-23T21:23:22.196027+0000","last_active":"2024-04-23T21:23:22.302129+0000","last_peered":"2024-04-23T21:23:22.302129+0000","last_clean":"2024-04-23T21:23:22.302129+0000","last_became_active":"2024-04-23T21:23:22.193400+0000","last_became_peered":"2024-04-23T21:23:22.193400+0000","last_unstale":"2024-04-23T21:23:22.302129+0000","last_undegraded":"2024-04-23T21:23:22.302129+0000","last_fullsized":"2024-04-23T21:23:22.302129+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:27:08.531751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001229749,"stat_sum":{"num_bytes":11177774,"num_objects":30,"num_object_clones":1,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":21,"num_read":6023,"num_read_kb":40072,"num_write":1184,"num_write_kb":21661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"18"}]},{"pgid":"3.13","version":"133'901","reported_seq":3171,"reported_epoch":134,"state":"unknown","last_fresh":"2024-04-23T21:23:18.386889+0000","last_change":"2024-04-23T21:23:18.627616+0000","last_active":"2024-04-23T21:23:18.386889+0000","last_peered":"2024-04-23T21:23:17.490099+0000","last_clean":"2024-04-23T21:23:17.490099+0000","last_became_active":"2024-04-23T21:21:17.778760+0000","last_became_peered":"2024-04-23T21:21:17.778760+0000","last_unstale":"2024-04-23T21:23:18.386889+0000","last_undegraded":"2024-04-23T21:23:18.386889+0000","last_fullsized":"2024-04-23T21:23:18.386889+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024089210000000001,"stat_sum":{"num_bytes":15821178,"num_objects":32,"num_object_clones":3,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":22,"num_read":5486,"num_read_kb":39372,"num_write":1308,"num_write_kb":24122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"13","length":"16"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":805065972,"num_objects":1023,"num_object_clones":118,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":145,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":686,"num_read":184661,"num_read_kb":1007916,"num_write":19193,"num_write_kb":586999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":123547689,"num_keys_recovered":256,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26198016,"data_stored":166483986,"data_compressed":9343846,"data_compressed_allocated":11399168,"data_compressed_original":153816701,"omap_allocated":0,"internal_metadata":0},"log_size":16874,"ondisk_log_size":16874,"up":52,"acting":58,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":1,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":13,"acting":14,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42660,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329180,"statfs":{"total":96636764160,"available":96593080320,"internally_reserved":0,"allocated":3231744,"data_stored":7017844,"data_compressed":372769,"data_compressed_allocated":495616,"data_compressed_original":5333308,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48928,"kb_used_data":9424,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94322912,"statfs":{"total":96636764160,"available":96586661888,"internally_reserved":0,"allocated":9650176,"data_stored":82819704,"data_compressed":4663484,"data_compressed_allocated":5410816,"data_compressed_original":79580353,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69880,"kb_used_data":6760,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301960,"statfs":{"total":96636764160,"available":96565207040,"internally_reserved":0,"allocated":6922240,"data_stored":31332182,"data_compressed":1626970,"data_compressed_allocated":2072576,"data_compressed_original":27731617,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574877,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38896,"kb_used_data":3556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94332944,"statfs":{"total":96636764160,"available":96596934656,"internally_reserved":0,"allocated":3641344,"data_stored":8799106,"data_compressed":405966,"data_compressed_allocated":532480,"data_compressed_original":6786299,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528340,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":4664,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":4775936,"data_stored":15298062,"data_compressed":827165,"data_compressed_allocated":1056768,"data_compressed_original":12730368,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":55834574876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1368064,"data_stored":3876365,"data_compressed":187309,"data_compressed_allocated":286720,"data_compressed_original":2984952,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2834432,"data_stored":12396683,"data_compressed":727496,"data_compressed_allocated":917504,"data_compressed_original":10752000,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2035712,"data_stored":8023959,"data_compressed":405966,"data_compressed_allocated":532480,"data_compressed_original":6786299,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1019904,"data_stored":6676463,"data_compressed":354993,"data_compressed_allocated":483328,"data_compressed_original":6245965,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5201920,"data_stored":29222622,"data_compressed":1566835,"data_compressed_allocated":2011136,"data_compressed_original":26457761,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8482816,"data_stored":82270788,"data_compressed":4663484,"data_compressed_allocated":5410816,"data_compressed_original":79580353,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1622016,"data_stored":5823546,"data_compressed":352655,"data_compressed_allocated":479232,"data_compressed_original":4907324,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3633152,"data_stored":18193560,"data_compressed":1085108,"data_compressed_allocated":1277952,"data_compressed_original":16102047,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:23:24.139 INFO:tasks.thrashosds.thrasher:Setting 3.1f to [2, 3] 2024-04-23T21:23:24.140 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1f', '2', '3'] 2024-04-23T21:23:24.140 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1f 2 3 2024-04-23T21:23:24.140 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:24.134+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:24.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:24.235 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:24.235+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:24.261 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:141889 3.1d 3:ba10b1d1:::rbd_header.111828d511a8:head [call lock.lock in=76b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e140)", 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:23:24.262360+0000", 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 0.00056894499999999995, 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00059155999999999998, 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:24.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 141889 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262360+0000", 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262360+0000", 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262361+0000", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5400000000000001e-07 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262363+0000", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4779999999999998e-06 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262370+0000", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6889999999999998e-06 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:24.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262374+0000", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2309999999999998e-06 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262407+0000", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2379999999999998e-05 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262450+0000", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3659999999999999e-05 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:24.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262903+0000", 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00045237000000000001 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:24.262936+0000", 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2852000000000003e-05 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: ], 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 1 2024-04-23T21:23:24.266 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:24.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:24.306 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:24.307 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:24.327 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:24.327 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:24.344 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:24.344 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:24.357 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:24.358+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:24.361 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:24.361 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:24.379 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:24.379 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:24.396 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:24.397 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:24.416 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:24.417 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:24.417 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:24.417 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:24.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:24.426 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:24.436 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:24.436+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:24.537 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:24.538+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:24.568 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:24.569 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:24.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:24.638 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:24.638+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:24.721 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:24.721 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:24.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:24.739 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:24.739+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:24.830 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:24.830 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:24.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:24.841 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:24.841+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:24.942 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:24.942 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:24.943 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:24.943+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:24.952 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:25.043 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:25.044+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:25.054 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.054 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.063 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:25.147 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:25.148+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:25.195 INFO:teuthology.orchestra.run.smithi111.stderr:set 3.1f pg_upmap mapping to [2,3] 2024-04-23T21:23:25.206 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:23:25.210 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.210 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:25.254 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:25.254+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:25.347 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:25.348+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:25.368 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.369 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.377 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:25.448 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:25.449+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:25.482 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.482 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.491 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:25.548 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:25.548+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:25.592 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.592 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:25.649 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:25.649+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:25.743 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.743 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.749 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:25.749+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:25.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:25.851 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:25.851+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:25.893 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:25.893 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:25.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:25.985 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:25.985+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:26.042 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:26.043 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:26.051 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:26.053 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:26.054+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:26.154 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:26.155+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:26.196 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:23:26.197 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:26.197 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:26.208 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:26.254 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:26.254+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:26.317 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:26.317 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:26.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:26.368 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:26.368+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:26.422 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:26.466 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:26.466+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:26.473 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:26.473 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:26.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:26.544 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:26.544 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:26.557 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:26.559+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:26.561 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:26.561 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:26.578 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:26.579 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:26.595 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:26.596 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:26.613 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:26.614 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:26.614 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:26.630 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.39893518300001, 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.631 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.632 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:26.633 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.634 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.635 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.398905516, 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.636 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.637 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:26.638 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:26.639 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.398886108, 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.640 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.641 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.642 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.643 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.644 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.644 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:26.644 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:26.645 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.3988643, 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.646 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.647 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.648 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.649 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.650 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.398844697, 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.651 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.652 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.653 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:26.654 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.398825092, 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.655 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.656 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.657 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.658 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.398804251, 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.659 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:26.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:26.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:26.663 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.39877498600001, 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:26.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.39874379299999, 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.668 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:26.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.672 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.590660782, 2024-04-23T21:23:26.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:26.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587432563999997, 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.679 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587410183000003, 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.683 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:26.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.687 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587390905000007, 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.688 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:26.692 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587373471000006, 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:26.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587356021000005, 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.701 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587338646000006, 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:26.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:26.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:26.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:26.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587320603999999, 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:26.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.587295483000005, 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.711 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.713 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:26.715 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 98.585175910999993, 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:26.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.720 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 86.245552919999994, 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.721 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.724 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.725 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.725 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:26.725 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:26.725 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:26.726 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:26.726 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:26.727 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:26.727 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:26.728 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:26.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:26.729 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:26.727+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:26.827 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:26.827+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:26.870 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:26.870 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:26.871 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:26.871 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:26.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:26.963 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:26.963+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:27.021 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:27.021 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:27.022 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:27.022 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:27.022 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:27.028 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:27.029+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:27.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:27.129 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:27.130+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:27.174 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:27.174 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.534746258, 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.175 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:27.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:27.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.179 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.534714151, 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.180 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:27.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:27.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.183 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.53469076499999, 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:27.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:27.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:27.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:27.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.53467045799999, 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.188 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:27.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:27.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.534650646, 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:27.192 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:27.196 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533579248, 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.197 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:27.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533545903, 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.201 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.53351437400001, 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.205 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:27.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533486985, 2024-04-23T21:23:27.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:27.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533475079, 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.214 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533456835, 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.218 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:27.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:27.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533447018, 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533436771, 2024-04-23T21:23:27.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:27.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:27.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.231 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.53342712, 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:27.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.235 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.533393621, 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.236 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:27.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:27.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:27.239 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.529768197, 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.240 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.52975032800001, 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:27.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.529731599, 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.248 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:27.252 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.529713171, 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:27.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:27.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.256 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 100.529695266, 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:27.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:27.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:27.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:27.260 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:27.262 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:27.262 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:27.262+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:27.366 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:27.364+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:27.371 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:27.371 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:27.371 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:27.371 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:27.380 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:27.464 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:27.464+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:27.483 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:27.483 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:27.483 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:27.483 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:27.483 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:27.492 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:27.564 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:27.565+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:27.595 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639413549, 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.596 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:27.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639375429, 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:27.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:27.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:27.604 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:27.605 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.63935303, 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:27.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:27.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:27.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639333634, 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:27.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639314036, 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.614 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639293823, 2024-04-23T21:23:27.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.621 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.622 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.63926165700001, 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:27.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:27.626 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639237044, 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.627 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:27.631 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639216046, 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:27.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.634 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639195747, 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:27.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.639117958, 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:27.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:27.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.644 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.637622584, 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:27.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:27.649 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.637572694, 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:27.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.637548925, 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:27.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.63752791900001, 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:27.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.637508314, 2024-04-23T21:23:27.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:27.666 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.621506165, 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:27.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.62148765400001, 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.672 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 100.621467209, 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:27.675 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:27.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:27.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 95.642389996999995, 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:27.679 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:27.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:27.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:27.685 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:27.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:27.687 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:27.687+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:27.788 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:27.788+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:27.788 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:27.789 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:27.789 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:27.789 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:27.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:27.888 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:27.889+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:27.941 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:27.941 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:27.941 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:27.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:27.991 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:27.992+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:28.092 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:28.092+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440778547999997, 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.095 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:28.096 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:23:28.097 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.098 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:28.099 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440757423999997, 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.100 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.101 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.102 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.103 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440728156999995, 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.104 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.105 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.106 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.107 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.108 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440707395000004, 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.109 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.110 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.111 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.112 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.113 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440688369, 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.114 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:28.115 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.116 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.117 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:23:28.118 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.119 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440669529999994, 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.120 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.121 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.122 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.123 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440636127999994, 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.124 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:28.125 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.126 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.127 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:23:28.128 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440616382000002, 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.129 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.130 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.131 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.132 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440598279, 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.133 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.134 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:23:28.135 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:23:28.136 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:23:28.137 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440579077999999, 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.138 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.139 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.140 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.141 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.142 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440553679000004, 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.143 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.144 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.145 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.146 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.147 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440528553999997, 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.148 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.149 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.150 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.151 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440503543000005, 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.152 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.153 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.154 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.155 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440479973999999, 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:23:28.156 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.157 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.158 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.159 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440456819000005, 2024-04-23T21:23:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.161 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:28.162 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:28.163 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.164 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440436075999997, 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.165 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.166 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:28.167 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.168 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440417775, 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.169 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:28.170 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.171 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.172 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.173 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440400194999995, 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:28.174 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:28.175 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:28.176 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.177 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.440370672, 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.178 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.179 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.180 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:28.181 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 87.701115607999995, 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.182 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.183 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.184 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:28.185 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:28.186 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.187 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.188 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.188 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:28.189 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:28.195 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:28.195+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:28.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-04-23T21:23:28.296 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:28.297+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:28.306 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:28.306 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:28.306 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:28.306 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:28.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:28.397 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:28.397+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:28.459 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:28.460 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:28.460 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:28.460 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:28.460 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:28.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:28.498 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:28.498+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:28.571 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 83.444084661000005, 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:28.572 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.573 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:28.574 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.575 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:28.576 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.577 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.644088257999996, 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.578 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.579 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.580 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:28.581 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.644061738000005, 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:28.582 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.583 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.584 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.585 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.586 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.644041673999993, 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.587 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:28.588 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:28.589 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.590 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643749004, 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.591 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.592 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.593 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.594 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:28.595 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643739099000001, 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.596 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.597 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.598 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:28.599 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.600 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643436319000003, 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.601 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:28.602 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.605 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.606 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643399638999995, 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.607 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.608 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:28.609 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.610 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.611 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643378016, 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.612 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.613 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:28.614 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.615 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:28.616 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643357770999998, 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.617 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:28.618 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:28.619 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.620 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.643156454000007, 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.621 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:28.622 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.623 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.624 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642721782999999, 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.625 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:28.626 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:28.627 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:28.628 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642454958000002, 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.629 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.630 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.631 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.632 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.633 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642432756999995, 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.634 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.635 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.636 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.637 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:28.638 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642413168000004, 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.639 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.640 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.641 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642074855999994, 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.644 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:28.645 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.646 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.647 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:28.648 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642054274000003, 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.649 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:28.651 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.652 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.642035250000006, 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:28.653 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.654 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:28.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:28.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 69.641809085999995, 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:28.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:28.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:140851 3.12 3:4c3ea8ab:::rbd_header.11189a87d8f4:head [watch reconnect cookie 140596427927216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 15.228727251, 2024-04-23T21:23:28.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0289291229999999, 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 140851 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342804+0000", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.576e-06 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342809+0000", 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5969999999999999e-06 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342821+0000", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1286e-05 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342830+0000", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2560000000000008e-06 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342904+0000", 2024-04-23T21:23:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4219000000000002e-05 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342915+0000", 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1090999999999999e-05 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364226+0000", 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0213109119999999 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364289+0000", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3180000000000002e-05 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366791+0000", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025013290000000001 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366909+0000", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011882799999999999 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:28.667 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.371731+0000", 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048218489999999996 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:28.668 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:28.669 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:28.670 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:28.669+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:28.730 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:28.770 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:28.771+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:28.775 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:28.775 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:28.775 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:28.775 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:28.784 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:28.869 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:28.869 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:28.870 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:28.870+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:28.888 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:28.888 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:28.903 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:28.903 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:28.904 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:28.905 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:28.912 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:28.922 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:28.939 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:28.940 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:28.940 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:28.958 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:28.958 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:28.971 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:28.973+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:28.991 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:28.991 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:28.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:29.038 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.038 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.047 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:29.073 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:29.074+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:29.149 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.149 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.158 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:29.201 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is set 2024-04-23T21:23:29.202 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:29.202+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.270 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.273 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:29.274+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:29.280 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:29.374 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:29.374+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:29.382 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.383 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.391 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:29.475 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:29.476+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:29.535 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.535 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:29.585 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:29.585+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:29.677 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:29.677+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:29.686 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.687 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.695 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:29.780 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:29.780+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:29.838 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:29.838 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:29.847 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:29.888 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:29.888+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:29.980 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:29.980+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:30.178 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.178 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:30.179 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:30.178+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:30.212 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6] out_osds: [3, 0, 7] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:30.212 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:30.213 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T21:23:30.213 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-04-23T21:23:30.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-23T21:23:30.279 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:30.279+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:30.331 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.331 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.341 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:30.382 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:30.382+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:30.480 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:30.480+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:30.485 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.485 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.494 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:30.581 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:30.581+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:30.597 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.597 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.605 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:30.682 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:30.682+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:30.708 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.708 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.717 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:30.783 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:30.783+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:30.860 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.860 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.868 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:30.884 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:30.885+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:30.971 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:30.971 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:30.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:30.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:31.023 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:31.023+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:31.083 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:31.083 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:31.086 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:31.086+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:31.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:31.117 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:31.117 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:31.134 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:31.134 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:31.150 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:31.150 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:31.166 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:31.166 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:31.183 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:31.183 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:31.186 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:31.186+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:31.200 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:31.200 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:31.215 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-23T21:23:31.221 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-04-23T21:23:31.221 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:31.222 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:31.227 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.244 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589749588000004, 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:23:31.248 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589728464000004, 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.249 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589699197000002, 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.253 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:23:31.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589678434999996, 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.258 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.262 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589659409000006, 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:23:31.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.266 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.58964057, 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.267 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:23:31.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.271 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589607168000001, 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.272 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:23:31.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:31.276 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589587421999994, 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:31.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589569319000006, 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.281 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:23:31.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:23:31.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:23:31.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589550118000005, 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.285 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:23:31.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:23:31.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589524718999996, 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:23:31.289 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:23:31.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589499594000003, 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.297 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.298 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589474582999998, 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:23:31.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:23:31.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589451014000005, 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.303 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589427858999997, 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:31.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589407116000004, 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.315 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589388815000007, 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:31.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.320 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589371235000002, 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 99.589341712000007, 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.325 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:31.326 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:31.327 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.328 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 90.850086648000001, 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:31.329 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:31.330 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.331 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:31.332 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:31.333 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.334 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.335 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:31.336 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:31.337 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:31.336+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:31.437 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:31.437+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:31.481 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:31.481 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:31.481 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:31.481 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:31.489 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:31.538 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:31.539+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:31.591 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:31.591 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:31.591 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:31.591 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:31.592 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:31.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:31.643 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:31.643+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.703 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 86.575670735000003, 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.704 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.705 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775674331999994, 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.709 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:31.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.711 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775647812000003, 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.714 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.718 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775627748000005, 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:31.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:31.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:31.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775335077999998, 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.723 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.727 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775325172999999, 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.730 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.731 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.775022393, 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:31.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:31.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:31.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:31.735 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774985713000007, 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:31.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:31.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:31.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:31.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774964089999997, 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.740 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.742 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.743 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774943844999996, 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774742528000004, 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.748 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:31.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774307856999997, 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:31.752 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.756 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774041032, 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.758 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.759 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.760 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.774018831000006, 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.761 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.764 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.773999242000002, 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:31.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:31.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.773660930000005, 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:31.769 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.773 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.773640348000001, 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.775 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.776 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.777 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.773621324000004, 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.778 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.779 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.780 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.781 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 72.773395160000007, 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.782 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.783 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:31.784 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:31.785 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:140851 3.12 3:4c3ea8ab:::rbd_header.11189a87d8f4:head [watch reconnect cookie 140596427927216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 18.360313325, 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0289291229999999, 2024-04-23T21:23:31.786 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 140851 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.787 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342804+0000", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.576e-06 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342809+0000", 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5969999999999999e-06 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.788 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342821+0000", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1286e-05 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342830+0000", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2560000000000008e-06 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342904+0000", 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4219000000000002e-05 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.789 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342915+0000", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1090999999999999e-05 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364226+0000", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0213109119999999 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364289+0000", 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3180000000000002e-05 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.790 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366791+0000", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025013290000000001 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366909+0000", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011882799999999999 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.371731+0000", 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048218489999999996 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.792 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:31.792 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:31.792 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:31.793 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:31.794 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:31.794+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:31.894 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:31.894+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:31.897 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:31.897 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:31.897 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:31.897 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:31.905 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:31.994 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:31.995+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:32.048 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.048 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:32.048 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:32.048 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:32.048 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.057 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:32.095 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:32.095+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.159 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 103.218264304, 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.160 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:32.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388865374999995, 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.164 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.165 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.168 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388842440999994, 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:32.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.170 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388817122000006, 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.173 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:32.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.176 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388798402999996, 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.177 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.178 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.179 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388755721999999, 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.181 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:32.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388729581999996, 2024-04-23T21:23:32.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388711663999999, 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.190 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:32.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.194 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388680285000007, 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.198 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388663156000007, 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:32.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.202 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388634617999998, 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:32.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:32.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:32.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:32.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388591633000004, 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.207 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.210 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388564396000007, 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.211 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:32.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.215 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388540609000003, 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:32.219 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388515366999997, 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.220 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388496601, 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.224 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:32.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:32.228 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388439641000005, 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.231 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388417907000004, 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.236 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 78.388393686000001, 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:32.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.240 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 58.959124631000002, 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.241 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:32.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.247 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.249 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:32.249 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is set 2024-04-23T21:23:32.250 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:32.248+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:32.350 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:32.349+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:32.394 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.395 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:32.395 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:32.395 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:32.450 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:32.450+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:32.505 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.505 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:32.506 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:32.506 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:32.506 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.514 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:32.552 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:32.551+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:32.653 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:32.653+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:32.657 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.657 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6782 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7110656~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227590037, 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052409548, 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.658 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6782 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:32.659 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430344+0000", 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0800000000000001e-07 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.660 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430350+0000", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2410000000000003e-06 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430357+0000", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9700000000000002e-06 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430361+0000", 2024-04-23T21:23:32.661 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5650000000000003e-06 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482669+0000", 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052307971000000002 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482710+0000", 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1060000000000003e-05 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.662 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482754+0000", 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3433000000000002e-05 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6783 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 7831552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22756722600001, 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566319, 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.663 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6783 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.664 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430367+0000", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3599999999999999e-07 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430369+0000", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.747e-06 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430376+0000", 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8560000000000001e-06 2024-04-23T21:23:32.665 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430380+0000", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4909999999999999e-06 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482792+0000", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052412021000000003 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482882+0000", 2024-04-23T21:23:32.666 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9891999999999999e-05 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482933+0000", 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0976000000000001e-05 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6784 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227547616, 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.05266175, 2024-04-23T21:23:32.667 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6784 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.668 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430386+0000", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430387+0000", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2899999999999999e-07 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430389+0000", 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.841e-06 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.669 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430395+0000", 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0669999999999997e-06 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430399+0000", 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5310000000000003e-06 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.670 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.482966+0000", 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052566881000000003 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483007+0000", 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1264999999999999e-05 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483048+0000", 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0836e-05 2024-04-23T21:23:32.671 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6785 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227528753, 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052769727000000002, 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.672 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6785 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430405+0000", 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.673 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430406+0000", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5999999999999999e-07 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430408+0000", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9800000000000001e-06 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430414+0000", 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.674 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430418+0000", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483069+0000", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052651058000000001 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483114+0000", 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5148000000000002e-05 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.675 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483175+0000", 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0534e-05 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6786 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 8421376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227509458, 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052874828999999998, 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.676 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6786 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.677 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430425+0000", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4799999999999999e-07 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430427+0000", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0430000000000002e-06 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430440+0000", 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2605999999999999e-05 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.678 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430445+0000", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2009999999999998e-06 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483209+0000", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052764194 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483256+0000", 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6579999999999998e-05 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.679 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483299+0000", 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3856999999999997e-05 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6787 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9142272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227482897, 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052995397999999999, 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.680 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6787 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430451+0000", 2024-04-23T21:23:32.681 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430452+0000", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9700000000000002e-07 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430455+0000", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0249999999999998e-06 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430461+0000", 2024-04-23T21:23:32.682 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5590000000000002e-06 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430466+0000", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2159999999999997e-06 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483343+0000", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052876337000000002 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483393+0000", 2024-04-23T21:23:32.683 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9963999999999998e-05 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483447+0000", 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3900000000000002e-05 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6788 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9338880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22744166699999, 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053112032000000003, 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.684 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6788 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.685 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430492+0000", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430493+0000", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5599999999999995e-07 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430495+0000", 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5490000000000001e-06 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.686 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430503+0000", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4850000000000003e-06 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430508+0000", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5260000000000001e-06 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483489+0000", 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052980666000000003 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.687 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483560+0000", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0779999999999997e-05 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483604+0000", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4469999999999999e-05 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6789 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9535488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:32.688 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22740794000001, 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053193595000000003, 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6789 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.689 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430526+0000", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430527+0000", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.75e-07 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430529+0000", 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.385e-06 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.690 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430536+0000", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4449999999999999e-06 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430542+0000", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5890000000000002e-06 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483635+0000", 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053093431000000003 2024-04-23T21:23:32.691 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483681+0000", 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5636000000000001e-05 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483720+0000", 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8634000000000003e-05 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.692 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6790 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 9732096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227384531, 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053352472999999997, 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6790 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:32.693 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430550+0000", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430552+0000", 2024-04-23T21:23:32.694 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9259999999999999e-06 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430564+0000", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1902999999999999e-05 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430569+0000", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4210000000000003e-06 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483743+0000", 2024-04-23T21:23:32.695 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053173453000000002 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483822+0000", 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9626e-05 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483902+0000", 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9729999999999997e-05 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.696 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6791 3.1b 3:d978b474:::rbd_data.11181344e3fc.0000000000000000:8 [read 10452992~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227351281, 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053452089000000001, 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6791 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.697 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430583+0000", 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7100000000000002e-07 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.698 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430586+0000", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2689999999999998e-06 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430592+0000", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6239999999999996e-06 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430597+0000", 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0089999999999998e-06 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.699 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483922+0000", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053324612 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.483986+0000", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4696999999999994e-05 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484035+0000", 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8406999999999999e-05 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.700 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6792 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 229376~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.227327888, 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053576937999999998, 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.701 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6792 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430606+0000", 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.702 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430607+0000", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8599999999999999e-07 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430608+0000", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9439999999999999e-06 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430615+0000", 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3999999999999997e-06 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.703 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430621+0000", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484073+0000", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053451911999999997 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484120+0000", 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7447999999999997e-05 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.704 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484183+0000", 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3003000000000006e-05 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6793 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 425984~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22730733900001, 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053717952999999999, 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.705 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6793 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:32.706 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430627+0000", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4499999999999998e-07 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430629+0000", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6750000000000001e-06 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430641+0000", 2024-04-23T21:23:32.707 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2697999999999999e-05 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430646+0000", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-06 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484228+0000", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053581469 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484281+0000", 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3019000000000002e-05 2024-04-23T21:23:32.708 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484345+0000", 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3726999999999995e-05 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6794 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 622592~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22728157900001, 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053854880000000001, 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.709 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6794 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.430652+0000", 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.710 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431445+0000", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00079274200000000003 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431458+0000", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2952000000000001e-05 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431470+0000", 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1969e-05 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.711 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431478+0000", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5739999999999998e-06 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484378+0000", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052900444999999997 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484447+0000", 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8503000000000005e-05 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.712 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484507+0000", 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0695000000000001e-05 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6795 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 819200~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.226443563, 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053176113999999997, 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.713 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6795 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431490+0000", 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.714 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431491+0000", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3000000000000001e-07 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431493+0000", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4710000000000001e-06 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431501+0000", 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2259999999999999e-06 2024-04-23T21:23:32.715 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431506+0000", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2240000000000003e-06 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484540+0000", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053033673000000003 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484597+0000", 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7210000000000003e-05 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.716 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484667+0000", 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9779999999999999e-05 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6796 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1540096~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.226420232, 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.053281372, 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.717 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6796 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.431514+0000", 2024-04-23T21:23:32.718 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432196+0000", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00068173500000000004 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432201+0000", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1229999999999999e-06 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432374+0000", 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017341899999999999 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.719 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432385+0000", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1243000000000001e-05 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484696+0000", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052310621000000002 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484736+0000", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0235999999999999e-05 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.720 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484795+0000", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8995e-05 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6797 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1736704~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.225532939, 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052534275999999998, 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.721 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6797 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432401+0000", 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.722 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432605+0000", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00020384400000000001 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432610+0000", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0340000000000004e-06 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432620+0000", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0479000000000001e-05 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.723 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432625+0000", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.707e-06 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484825+0000", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052199779000000002 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484874+0000", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9333999999999999e-05 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.724 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484935+0000", 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1099000000000006e-05 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6798 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 1933312~32768] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.225299359, 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425591000000001, 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.725 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6798 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432635+0000", 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.726 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432727+0000", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1828000000000002e-05 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432732+0000", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3260000000000003e-06 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432739+0000", 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.0910000000000004e-06 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.727 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432743+0000", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3109999999999999e-06 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.484961+0000", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052217995000000003 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485003+0000", 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1344000000000003e-05 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.728 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485060+0000", 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7695999999999998e-05 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6799 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 2129920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22518266100001, 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052420640999999997, 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.729 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6799 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.432751+0000", 2024-04-23T21:23:32.730 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433064+0000", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00031291199999999999 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433070+0000", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1009999999999997e-06 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433081+0000", 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0905e-05 2024-04-23T21:23:32.731 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433088+0000", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9759999999999998e-06 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485086+0000", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.051997451 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485126+0000", 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0578999999999999e-05 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.732 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485172+0000", 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5717000000000003e-05 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6801 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 3047424~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.224808247, 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052425017999999997, 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.733 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6801 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433126+0000", 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.734 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433344+0000", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00021807099999999999 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433349+0000", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7049999999999998e-06 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433369+0000", 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.0046000000000001e-05 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.735 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.433376+0000", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.34e-06 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485463+0000", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052087381000000002 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485507+0000", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3961000000000001e-05 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.736 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.485551+0000", 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3513999999999997e-05 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:6809 3.1b 3:da5092c7:::rbd_data.11181344e3fc.0000000000000001:8 [read 5668864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e27)", 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 123.22345136200001, 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052349623999999997, 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:32.737 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 6809 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434483+0000", 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.738 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434646+0000", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016305399999999999 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434652+0000", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1720000000000004e-06 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434661+0000", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7520000000000002e-06 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.434666+0000", 2024-04-23T21:23:32.739 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486740+0000", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.052074038000000003 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486790+0000", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9836999999999997e-05 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:29.486832+0000", 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2775999999999998e-05 2024-04-23T21:23:32.740 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:32.741 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:32.741 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.742 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:32.783 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:32.783+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:32.855 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:32.856+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:32.884 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:32.885 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:32.885 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:32.885 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:32.893 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:32.977 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:32.977+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:33.035 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:33.035 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:33.036 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:33.036 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:33.036 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:33.044 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:33.057 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:33.058+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:33.158 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:33.159+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.188 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957371753, 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:33.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:33.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957342086, 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.193 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.194 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957322678, 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.197 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.199 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.95730087, 2024-04-23T21:23:33.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.205 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957281267, 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.95726166199999, 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.210 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.212 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.213 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957240821, 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:33.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:33.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:33.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:33.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.957211556, 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.218 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 106.95718036300001, 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.222 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:33.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:33.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:33.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.149097352, 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:33.226 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.227 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.228 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:33.229 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.230 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.14586913399999, 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.231 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.232 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.233 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.234 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145846753, 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.235 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.236 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.237 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:33.238 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145827475, 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.239 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.240 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.241 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:33.242 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145810041, 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:33.243 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.244 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.245 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.246 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.247 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145792591, 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.248 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.249 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.250 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.251 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145775216, 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:33.252 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.253 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.254 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.255 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145757174, 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.256 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:33.257 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:33.258 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:33.259 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.145732053, 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:33.260 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.261 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.262 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.263 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:33.264 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 105.14361248100001, 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:33.265 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:33.266 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:33.267 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.268 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 92.803989490000006, 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.269 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:33.270 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.271 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.272 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.273 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:33.273 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:33.273 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:33.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-23T21:23:33.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:33.275 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:33.274+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:33.375 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:33.376+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:33.421 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:33.421 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:33.421 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:33.422 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:33.431 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:33.436 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:33.436 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:33.454 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:33.454 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:33.476 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:33.477+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:33.487 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:33.487 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:33.504 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:33.504 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:33.537 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:33.537 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:33.554 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:33.555 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:33.557 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:33.557 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:33.566 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:33.572 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:33.573 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:33.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:33.578 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:33.578+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:33.675 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:33.676 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:33.679 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:33.679+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:33.684 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:33.780 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:33.780+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:33.788 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:33.788 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:33.798 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:33.880 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:33.881+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:33.900 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:33.900 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:33.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:33.981 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:33.981+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:34.011 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.011 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.019 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:34.082 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:34.082+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:34.122 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.123 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.131 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:34.183 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:34.184+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:34.250 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:23:34.284 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.284 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.294 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:34.294+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:34.295 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:34.384 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:34.385+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:34.406 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.406 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.415 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:34.486 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:34.487+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:34.518 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.519 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.527 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:34.587 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:34.588+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:34.670 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.670 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.679 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:34.687 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:34.688+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:34.781 INFO:teuthology.orchestra.run.smithi192.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.781 INFO:teuthology.orchestra.run.smithi192.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.788 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:34.788+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:34.790 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:34.889 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:34.889+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:34.892 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:34.892 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:34.901 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:34.990 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-04-23T21:23:34.990+0000 7f3c786ca640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 32389) UID: 0 2024-04-23T21:23:35.003 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.003 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.011 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:35.091 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:35.091+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:35.113 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.113 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.122 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-04-23T21:23:35.191 INFO:tasks.ceph.osd.0.smithi111.stderr:2024-04-23T21:23:35.191+0000 7f883341c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 32220) UID: 0 2024-04-23T21:23:35.225 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.225 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-04-23T21:23:35.244 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:23:35.292 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:35.293+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:35.358 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.358 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-04-23T21:23:35.412 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:35.412+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:35.471 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.471 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.480 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-04-23T21:23:35.521 INFO:tasks.ceph.osd.2.smithi111.stderr:2024-04-23T21:23:35.522+0000 7fe276d13640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 32230) UID: 0 2024-04-23T21:23:35.577 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-23T21:23:35.595 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:35.596+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:35.625 INFO:teuthology.orchestra.run.smithi111.stdout:ERROR: (22) Invalid argument 2024-04-23T21:23:35.625 INFO:teuthology.orchestra.run.smithi111.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-23T21:23:35.634 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-04-23T21:23:35.695 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-04-23T21:23:35.696+0000 7fd109ab1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 32388) UID: 0 2024-04-23T21:23:35.700 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0: {} 2024-04-23T21:23:35.701 INFO:teuthology.orchestra.run.smithi111.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-23T21:23:35.718 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1: {} 2024-04-23T21:23:35.719 INFO:teuthology.orchestra.run.smithi111.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-23T21:23:35.736 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2: {} 2024-04-23T21:23:35.736 INFO:teuthology.orchestra.run.smithi111.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-23T21:23:35.741 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:35.741 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:35.741 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:35.741 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:35.741 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:35.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-04-23T21:23:35.753 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3: {} 2024-04-23T21:23:35.753 INFO:teuthology.orchestra.run.smithi111.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-23T21:23:35.771 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4: {} 2024-04-23T21:23:35.771 INFO:teuthology.orchestra.run.smithi111.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-23T21:23:35.771 INFO:teuthology.orchestra.run.smithi111.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-04-23T21:23:35.789 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6: {} 2024-04-23T21:23:35.789 INFO:teuthology.orchestra.run.smithi111.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-23T21:23:35.796 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:35.797+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:35.806 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7: {} 2024-04-23T21:23:35.807 INFO:teuthology.orchestra.run.smithi111.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-23T21:23:35.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-23T21:23:35.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-23T21:23:35.855 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28527 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1243136~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624585772, 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010320411999999999, 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.856 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28527 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.857 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.858 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230951+0000", 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230953+0000", 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9089999999999998e-06 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.859 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230968+0000", 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4548e-05 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230973+0000", 2024-04-23T21:23:35.860 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4790000000000004e-06 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241067+0000", 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01009354 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.861 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241143+0000", 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6460999999999999e-05 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.862 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241271+0000", 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012810799999999999 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.863 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28528 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1265664~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624556105, 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010625594, 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.864 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28528 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.865 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:35.866 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.867 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.867 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.867 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.867 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230981+0000", 2024-04-23T21:23:35.867 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9599999999999999e-07 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230983+0000", 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2620000000000001e-06 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.868 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230989+0000", 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7989999999999999e-06 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.230994+0000", 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7180000000000004e-06 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.869 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241300+0000", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010306137 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241430+0000", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013044599999999999 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241606+0000", 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00017573600000000001 2024-04-23T21:23:35.870 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28532 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1288192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624536697, 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010821457, 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.871 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28532 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.872 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231000+0000", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231002+0000", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.03e-06 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231010+0000", 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.7810000000000005e-06 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.873 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231015+0000", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241638+0000", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.01062216 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241725+0000", 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.7027999999999994e-05 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.874 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241821+0000", 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6821e-05 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28537 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1310720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624514889, 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010974037000000001, 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.875 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28537 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:35.876 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231022+0000", 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231024+0000", 2024-04-23T21:23:35.877 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1390000000000002e-06 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231030+0000", 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231035+0000", 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7809999999999996e-06 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.878 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241849+0000", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010814067 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241908+0000", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8860999999999998e-05 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.241996+0000", 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8109999999999995e-05 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.879 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28544 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1333248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624495286, 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011242815999999999, 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28544 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.880 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231041+0000", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231042+0000", 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:35.881 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231044+0000", 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9829999999999999e-06 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231050+0000", 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.913e-06 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.882 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231054+0000", 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4569999999999998e-06 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242023+0000", 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010969044000000001 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242095+0000", 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1724999999999995e-05 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.883 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242284+0000", 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00018928900000000001 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28550 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1355776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.62447568100001, 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011494581, 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.884 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28550 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:35.885 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231061+0000", 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231063+0000", 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.897e-06 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.886 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231071+0000", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5399999999999998e-06 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231075+0000", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5589999999999999e-06 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242315+0000", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011239717 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.887 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242398+0000", 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3035999999999996e-05 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242556+0000", 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015743699999999999 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28554 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1378304~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:35.888 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.62445484, 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011750142999999999, 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28554 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:35.889 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231082+0000", 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5199999999999998e-07 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.890 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231084+0000", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9929999999999998e-06 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231092+0000", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9789999999999993e-06 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231104+0000", 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1734e-05 2024-04-23T21:23:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242586+0000", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011481847 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242667+0000", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1003999999999999e-05 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242832+0000", 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165234 2024-04-23T21:23:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28562 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1400832~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.624425575, 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011983321999999999, 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.893 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28562 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.894 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231111+0000", 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.895 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231114+0000", 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1349999999999999e-06 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231120+0000", 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0730000000000001e-06 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.896 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231124+0000", 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7729999999999999e-06 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242861+0000", 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011736229000000001 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.897 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.242941+0000", 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0006999999999995e-05 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243094+0000", 2024-04-23T21:23:35.898 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001537 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:28572 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 1423360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.62439438200001, 2024-04-23T21:23:35.899 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012226818, 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 28572 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.900 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.901 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231142+0000", 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231143+0000", 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1399999999999997e-07 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.902 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.905 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.905 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231145+0000", 2024-04-23T21:23:35.905 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3920000000000001e-06 2024-04-23T21:23:35.905 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.905 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231152+0000", 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9630000000000001e-06 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.906 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.231158+0000", 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4160000000000003e-06 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243124+0000", 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011966133 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.907 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243210+0000", 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6508000000000006e-05 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.243369+0000", 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015879199999999999 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.908 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45216 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 100352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.816311371, 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010426592, 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.909 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45216 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.910 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039225+0000", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039226+0000", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0200000000000002e-07 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039228+0000", 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6189999999999998e-06 2024-04-23T21:23:35.911 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039235+0000", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9290000000000001e-06 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.039241+0000", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8429999999999998e-06 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049460+0000", 2024-04-23T21:23:35.912 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010218836 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049539+0000", 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.8919999999999997e-05 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.049652+0000", 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00011294300000000001 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.913 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45262 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 126976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.81308315299999, 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010358462000000001, 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45262 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.914 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042454+0000", 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:35.915 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042456+0000", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1710000000000001e-06 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042462+0000", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042466+0000", 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5870000000000003e-06 2024-04-23T21:23:35.916 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052618+0000", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010151323 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052690+0000", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1773999999999994e-05 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052812+0000", 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000122373 2024-04-23T21:23:35.917 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45264 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 129024~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.813060772, 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010558443000000001, 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.918 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45264 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.919 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042476+0000", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042478+0000", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.889e-06 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042485+0000", 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6950000000000003e-06 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.920 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042489+0000", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5560000000000001e-06 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052838+0000", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010348816 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.052914+0000", 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5588999999999997e-05 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.921 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053034+0000", 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0001205 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45268 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 0~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.813041494, 2024-04-23T21:23:35.922 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010792331, 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45268 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.923 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042495+0000", 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042496+0000", 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3500000000000002e-07 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.924 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042497+0000", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8789999999999999e-06 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042503+0000", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6509999999999998e-06 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042507+0000", 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4580000000000003e-06 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.925 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053068+0000", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010560287 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053143+0000", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5621e-05 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053287+0000", 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00014410000000000001 2024-04-23T21:23:35.926 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45275 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 4096~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.81302406, 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011017582, 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.927 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45275 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.928 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042513+0000", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.89e-07 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042515+0000", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.7710000000000001e-06 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042520+0000", 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1320000000000002e-06 2024-04-23T21:23:35.929 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042524+0000", 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4429999999999996e-06 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053315+0000", 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010790582999999999 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.930 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053391+0000", 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6217999999999994e-05 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053530+0000", 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00013904600000000001 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45278 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 6144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:35.931 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.81300661, 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011308346, 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45278 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.932 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042530+0000", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5499999999999999e-07 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042532+0000", 2024-04-23T21:23:35.933 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8059999999999999e-06 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042538+0000", 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6840000000000001e-06 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042542+0000", 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3989999999999997e-06 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.934 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053579+0000", 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011036504000000001 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053688+0000", 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000109073 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053838+0000", 2024-04-23T21:23:35.935 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015052500000000001 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45280 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 55296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.812989235, 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011553265, 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.936 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45280 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042547+0000", 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.937 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042548+0000", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042550+0000", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.748e-06 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042555+0000", 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5219999999999998e-06 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.938 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042560+0000", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9259999999999999e-06 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053869+0000", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011309071 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.053961+0000", 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1707999999999997e-05 2024-04-23T21:23:35.939 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054101+0000", 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000139937 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45291 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 49152~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.812971193, 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0117918, 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.940 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45291 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.941 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042565+0000", 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042566+0000", 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042573+0000", 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4549999999999998e-06 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.942 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042579+0000", 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8359999999999997e-06 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.943 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042583+0000", 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3930000000000001e-06 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054151+0000", 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011567396000000001 2024-04-23T21:23:35.944 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054232+0000", 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1463000000000003e-05 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.945 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.946 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.946 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054357+0000", 2024-04-23T21:23:35.946 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00012490999999999999 2024-04-23T21:23:35.946 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45293 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 51200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:35.947 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.812946072, 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012072248000000001, 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.948 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45293 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.949 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:35.950 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.950 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.950 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.950 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.950 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042591+0000", 2024-04-23T21:23:35.951 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4200000000000001e-07 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042599+0000", 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5490000000000001e-06 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.952 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042604+0000", 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4489999999999998e-06 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.953 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.042608+0000", 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3309999999999997e-06 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.954 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054398+0000", 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.011790124000000001 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054498+0000", 2024-04-23T21:23:35.955 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9270000000000003e-05 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054663+0000", 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000165083 2024-04-23T21:23:35.956 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:45306 3.0 3:063ee5cc:::rbd_data.111825873f8b.0000000000000008:12 [read 2048~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44)", 2024-04-23T21:23:35.957 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 107.8108265, 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010477386999999999, 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 45306 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.960 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044710+0000", 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044711+0000", 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4499999999999997e-07 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044719+0000", 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3469999999999999e-06 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.961 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044727+0000", 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0309999999999998e-06 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.044733+0000", 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9560000000000002e-06 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.962 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.054939+0000", 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.010206338000000001 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055037+0000", 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7598999999999998e-05 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.963 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.055188+0000", 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015057100000000001 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_repop(client.4376.0:61142 3.4 e56/48 3:2140a2a5:::rbd_data.1118e08f6c14.0000000000000000:head v 56'150, mlcod=55'148)", 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 95.471203509000006, 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.012786384, 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:35.964 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384333+0000", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.384334+0000", 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.088e-06 2024-04-23T21:23:35.965 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390060+0000", 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0057252689999999998 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390078+0000", 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8218000000000001e-05 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:35.966 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390088+0000", 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0169e-05 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390151+0000", 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2984000000000005e-05 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.967 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.390175+0000", 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4408000000000001e-05 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397059+0000", 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0068838299999999996 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:35.968 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:00.397120+0000", 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0418000000000003e-05 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:35.969 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:35.971 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-04-23T21:23:35.972 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:35.971+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:36.072 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-04-23T21:23:36.072+0000 7f3240519640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 32382) UID: 0 2024-04-23T21:23:36.074 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:36.074 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:36.074 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:36.074 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:36.083 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-04-23T21:23:36.173 INFO:tasks.ceph.osd.1.smithi111.stderr:2024-04-23T21:23:36.174+0000 7f4a0dad8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 42573) UID: 0 2024-04-23T21:23:36.184 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:36.184 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:36.184 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:36.184 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:36.184 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:36.193 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-04-23T21:23:36.233 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6] out_osds: [3, 0, 7] dead_osds: [5] live_osds: [0, 2, 3, 4, 6, 7, 1] 2024-04-23T21:23:36.233 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-23T21:23:36.234 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-23T21:23:36.234 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.792847 2024-04-23T21:23:36.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0.7928470830465169 2024-04-23T21:23:36.278 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:36.279+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:36.350 INFO:teuthology.orchestra.run.smithi111.stdout:All operations completed A-OK! 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36227 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2220032~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.351 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.71114228099999, 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028698669, 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36227 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.352 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639755+0000", 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.353 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639757+0000", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1260000000000001e-06 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639764+0000", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.083e-06 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639768+0000", 2024-04-23T21:23:36.354 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7979999999999996e-06 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667888+0000", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028119567000000002 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.667960+0000", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2473999999999998e-05 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668454+0000", 2024-04-23T21:23:36.355 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00049311999999999999 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36228 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2836480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.711110174, 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028447777, 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.356 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36228 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.357 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639787+0000", 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.3000000000000001e-07 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.358 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639790+0000", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4420000000000001e-06 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639797+0000", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9269999999999999e-06 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639802+0000", 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4770000000000002e-06 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.359 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668070+0000", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028268043999999999 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668154+0000", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3349000000000004e-05 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668235+0000", 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.360 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36229 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2242560~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.711086788, 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028614415000000001, 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36229 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.361 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639810+0000", 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.362 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639811+0000", 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7899999999999999e-07 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639813+0000", 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2680000000000001e-06 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.363 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639819+0000", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0129999999999999e-06 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639824+0000", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6040000000000003e-06 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668258+0000", 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028434653000000001 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.364 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668331+0000", 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.2793000000000006e-05 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668425+0000", 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3604999999999996e-05 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.365 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36230 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2859008~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.711066481, 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028795184000000001, 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36230 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.366 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639831+0000", 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5300000000000004e-07 2024-04-23T21:23:36.367 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639833+0000", 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2529999999999999e-06 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639839+0000", 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8420000000000001e-06 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.368 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639844+0000", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.42e-06 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668466+0000", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028622786000000001 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668540+0000", 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.3904000000000003e-05 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.369 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668626+0000", 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5426000000000005e-05 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36231 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2265088~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.711046669, 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029251049000000001, 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.370 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36231 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.639850+0000", 2024-04-23T21:23:36.371 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640445+0000", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00059411000000000002 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640448+0000", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8709999999999999e-06 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640458+0000", 2024-04-23T21:23:36.372 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7589999999999992e-06 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640465+0000", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4269999999999996e-06 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.668840+0000", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028375834999999999 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669006+0000", 2024-04-23T21:23:36.373 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00016599299999999999 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669101+0000", 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.5054000000000002e-05 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36232 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2881536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:36.374 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709975271, 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028524420000000002, 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36232 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.375 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640922+0000", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640923+0000", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1299999999999999e-07 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640927+0000", 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0500000000000002e-06 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.376 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640938+0000", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1136e-05 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640944+0000", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.5819999999999998e-06 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669227+0000", 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028282525999999999 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.377 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669303+0000", 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6249999999999997e-05 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669446+0000", 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000143163 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.378 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36233 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2287616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709941926, 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028675330999999998, 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36233 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.379 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640955+0000", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640956+0000", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.380 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640958+0000", 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6029999999999999e-06 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640965+0000", 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.781e-06 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640977+0000", 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2234e-05 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.381 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669476+0000", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028498684999999999 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669548+0000", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1511e-05 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669631+0000", 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2892999999999994e-05 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.382 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36237 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2310144~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709910397, 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028827050999999999, 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36237 2024-04-23T21:23:36.383 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640987+0000", 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.1699999999999998e-07 2024-04-23T21:23:36.384 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640990+0000", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4329999999999998e-06 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.640996+0000", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.404e-06 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641008+0000", 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.2344000000000001e-05 2024-04-23T21:23:36.385 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669660+0000", 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028651473 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669728+0000", 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.8079999999999999e-05 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.386 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669814+0000", 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5799999999999998e-05 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36239 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2332672~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70988300800001, 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029030147999999999, 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.387 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36239 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:36.388 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641014+0000", 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641016+0000", 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.638e-06 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.389 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641020+0000", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8330000000000004e-06 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641023+0000", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.6309999999999999e-06 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669869+0000", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028846915000000001 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.390 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.669951+0000", 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1919e-05 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670044+0000", 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.2905999999999994e-05 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36241 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2355200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:36.391 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70987110199999, 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029258821000000001, 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36241 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.392 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641026+0000", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1e-07 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641035+0000", 2024-04-23T21:23:36.393 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5280000000000003e-06 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641038+0000", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.4319999999999999e-06 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641041+0000", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4119999999999999e-06 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670074+0000", 2024-04-23T21:23:36.394 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029033171999999999 2024-04-23T21:23:36.395 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.395 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.395 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.395 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670191+0000", 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000117646 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670285+0000", 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3420999999999999e-05 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.397 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36243 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2377728~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709852858, 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029439916, 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.398 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36243 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.399 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641044+0000", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2999999999999999e-07 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641046+0000", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1799999999999999e-06 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641049+0000", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9560000000000002e-06 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.400 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641051+0000", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3520000000000001e-06 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670314+0000", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029263010999999998 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670389+0000", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5308000000000005e-05 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.401 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670484+0000", 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4878999999999997e-05 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36245 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2400256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709843041, 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029615572999999999, 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.402 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36245 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.403 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641054+0000", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641055+0000", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1299999999999998e-07 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641058+0000", 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8140000000000001e-06 2024-04-23T21:23:36.404 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641060+0000", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3159999999999999e-06 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670513+0000", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029452539999999999 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670589+0000", 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.5638999999999999e-05 2024-04-23T21:23:36.405 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670670+0000", 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.1107999999999997e-05 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36247 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2422784~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70983279399999, 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.0299543, 2024-04-23T21:23:36.406 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36247 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.407 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641064+0000", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641065+0000", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.03e-07 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641066+0000", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.6599999999999994e-07 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.408 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641068+0000", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.8059999999999999e-06 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641071+0000", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2910000000000002e-06 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670856+0000", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029785565 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.409 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.670933+0000", 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.6737999999999996e-05 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671019+0000", 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5630999999999994e-05 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36249 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2445312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:36.410 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70982314299999, 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030120589, 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36249 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:36.411 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641074+0000", 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.4299999999999999e-07 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.412 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641075+0000", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0720000000000001e-06 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641078+0000", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.7860000000000001e-06 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641080+0000", 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.351e-06 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.413 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671046+0000", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029965855 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671116+0000", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9246999999999996e-05 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671195+0000", 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9035000000000005e-05 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.414 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36251 3.1e 3:7eb6c9f2:::rbd_data.111825873f8b.0000000000000002:12 [read 2467840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.709789644, 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030253677, 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36251 2024-04-23T21:23:36.415 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641107+0000", 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.416 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641108+0000", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.08e-07 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641109+0000", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.4729999999999999e-06 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641113+0000", 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.5300000000000001e-06 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.417 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.641116+0000", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3510000000000002e-06 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671220+0000", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.030103524 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671286+0000", 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6618000000000005e-05 2024-04-23T21:23:36.418 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.671361+0000", 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4772999999999997e-05 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36366 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2187264~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70616422000001, 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028357276000000001, 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.419 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36366 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.420 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644733+0000", 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.53e-07 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644735+0000", 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.8530000000000001e-06 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.421 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644741+0000", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644745+0000", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1779999999999996e-06 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.672919+0000", 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028174347999999998 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.422 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673002+0000", 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.2460000000000007e-05 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673090+0000", 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8381999999999995e-05 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36368 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2209792~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.423 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.706146351, 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028601089999999999, 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36368 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:36.424 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644751+0000", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644753+0000", 2024-04-23T21:23:36.425 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.846e-06 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644759+0000", 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.4099999999999996e-06 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644764+0000", 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4290000000000003e-06 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.426 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673119+0000", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028355222999999999 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673194+0000", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4727999999999993e-05 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673352+0000", 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015809099999999999 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.427 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36370 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2232320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.706127622, 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028770627999999999, 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36370 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.428 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644769+0000", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644770+0000", 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.6399999999999998e-07 2024-04-23T21:23:36.429 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644772+0000", 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.778e-06 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644778+0000", 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.229e-06 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.430 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644782+0000", 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4100000000000001e-06 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673380+0000", 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028597406999999998 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673452+0000", 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.216e-05 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.431 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673540+0000", 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8280000000000002e-05 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36372 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2254848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.70610919400001, 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028924115, 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.432 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36372 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.433 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644788+0000", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.3299999999999998e-07 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644790+0000", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1289999999999999e-06 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644796+0000", 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6890000000000001e-06 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.434 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644800+0000", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2100000000000003e-06 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673565+0000", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.028765199000000002 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673632+0000", 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6526000000000006e-05 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.435 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673712+0000", 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0029000000000002e-05 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:36375 3.2 3:417dd450:::rbd_data.111825873f8b.0000000000000005:12 [read 2277376~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.706091289, 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.029078739999999999, 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:36.436 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 36375 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.644806+0000", 2024-04-23T21:23:36.437 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646458+0000", 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.001652021 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646462+0000", 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.262e-06 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:36.438 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646471+0000", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.5769999999999994e-06 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.646485+0000", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.416e-05 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673737+0000", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.027252460999999999 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:36.439 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673804+0000", 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6282e-05 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:46.673885+0000", 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0976999999999994e-05 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:36.440 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:36.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-04-23T21:23:36.443 INFO:tasks.ceph.osd.3.smithi111.stderr:2024-04-23T21:23:36.443+0000 7ff3d79d8640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 32241) UID: 0 2024-04-23T21:23:36.443 DEBUG:teuthology.parallel:result is None 2024-04-23T21:23:36.444 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-04-23T21:23:36.455 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-23T21:23:36.493 INFO:tasks.thrashosds:joining thrashosds 2024-04-23T21:23:36.596 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:36.596 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:36.596 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:36.596 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:36.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-04-23T21:23:36.707 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:36.707 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:36.707 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:36.716 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-04-23T21:23:36.820 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:36.820 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40031 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1599488~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:36.821 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863405617, 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056236508999999997, 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.822 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40031 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.823 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:36.824 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956241+0000", 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.8100000000000002e-07 2024-04-23T21:23:36.825 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956245+0000", 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7799999999999998e-06 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.826 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956259+0000", 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3675999999999999e-05 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.827 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956267+0000", 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.6789999999999994e-06 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012293+0000", 2024-04-23T21:23:36.828 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056026131 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012380+0000", 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7009999999999995e-05 2024-04-23T21:23:36.829 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012477+0000", 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6552000000000007e-05 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.830 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40033 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1622016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:36.831 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863367497, 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056466479999999999, 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40033 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.832 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956279+0000", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3499999999999996e-07 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956282+0000", 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.419e-06 2024-04-23T21:23:36.833 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956289+0000", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1760000000000004e-06 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956294+0000", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4639999999999997e-06 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012551+0000", 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056256913999999998 2024-04-23T21:23:36.834 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012648+0000", 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.6340000000000003e-05 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012745+0000", 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7532000000000001e-05 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.835 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40035 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1644544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863345098, 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056710729000000001, 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40035 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.836 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956301+0000", 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6899999999999998e-07 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.837 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956303+0000", 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0449999999999999e-06 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956309+0000", 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9540000000000001e-06 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.838 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956314+0000", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6569999999999996e-06 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012803+0000", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056488717000000001 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.012894+0000", 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1728e-05 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.839 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013012+0000", 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000117259 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40037 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1667072~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863325702, 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056832527000000001, 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.840 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40037 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956320+0000", 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.841 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956321+0000", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7e-07 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956323+0000", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1129999999999999e-06 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956329+0000", 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0260000000000004e-06 2024-04-23T21:23:36.842 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956334+0000", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0760000000000002e-06 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013060+0000", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056725949999999997 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013103+0000", 2024-04-23T21:23:36.843 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2744000000000003e-05 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013153+0000", 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0247999999999997e-05 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40039 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1689600~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863306104, 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056916096999999999, 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.844 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40039 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.845 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956340+0000", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.84e-07 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956342+0000", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9199999999999998e-06 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.846 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956348+0000", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8610000000000003e-06 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956354+0000", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4659999999999998e-06 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013170+0000", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056816058000000003 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013207+0000", 2024-04-23T21:23:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7218999999999997e-05 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013256+0000", 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9188999999999999e-05 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40041 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1712128~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863285891, 2024-04-23T21:23:36.848 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057014793000000001, 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40041 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.849 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956360+0000", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956361+0000", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2100000000000002e-07 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956371+0000", 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0098000000000001e-05 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.850 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956379+0000", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4290000000000001e-06 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956385+0000", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5849999999999999e-06 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013273+0000", 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056887859999999998 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.851 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013321+0000", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7889999999999997e-05 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013375+0000", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4509999999999998e-05 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40043 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1734656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:36.852 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.86325372500001, 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057109104000000001, 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40043 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:36.853 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956392+0000", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956393+0000", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0100000000000005e-07 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956395+0000", 2024-04-23T21:23:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.261e-06 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956401+0000", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3110000000000002e-06 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956407+0000", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0320000000000003e-06 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013397+0000", 2024-04-23T21:23:36.855 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056990804999999999 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013446+0000", 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8999000000000003e-05 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013502+0000", 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5195000000000003e-05 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.856 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40045 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1757184~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863229112, 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057198735000000001, 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40045 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.857 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956417+0000", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956418+0000", 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9599999999999999e-07 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.858 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956420+0000", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.272e-06 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956427+0000", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6810000000000001e-06 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956431+0000", 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7770000000000002e-06 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.859 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013517+0000", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057085521 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013565+0000", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7885e-05 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013616+0000", 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1002999999999999e-05 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.860 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40047 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1779712~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863208114, 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.05729124, 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40047 2024-04-23T21:23:36.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956438+0000", 2024-04-23T21:23:36.862 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956440+0000", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0619999999999999e-06 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956447+0000", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7889999999999997e-06 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956452+0000", 2024-04-23T21:23:36.863 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.617e-06 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013631+0000", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057178899999999998 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013679+0000", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8228e-05 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013729+0000", 2024-04-23T21:23:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0260999999999999e-05 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40048 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1802240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863187815, 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057401473000000001, 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.865 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40048 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956458+0000", 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.866 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956508+0000", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9518000000000003e-05 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956511+0000", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8550000000000002e-06 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956520+0000", 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0499999999999997e-06 2024-04-23T21:23:36.867 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956527+0000", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1280000000000002e-06 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013744+0000", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057217294000000002 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013793+0000", 2024-04-23T21:23:36.868 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8776000000000002e-05 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013860+0000", 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6852000000000003e-05 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40050 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1824768~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.863110026, 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.057456038000000001, 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.869 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40050 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.870 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.956536+0000", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957932+0000", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0013962340000000001 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.871 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957937+0000", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7330000000000003e-06 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957947+0000", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0397999999999999e-05 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.957955+0000", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.413e-06 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.872 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013875+0000", 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.055920419999999998 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013922+0000", 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6912000000000001e-05 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.013992+0000", 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9927999999999999e-05 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40054 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1869824~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.861614652, 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056215772999999997, 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40054 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.874 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958031+0000", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958032+0000", 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6700000000000003e-07 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.875 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958035+0000", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.2729999999999998e-06 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958056+0000", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0361999999999999e-05 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958067+0000", 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1504e-05 2024-04-23T21:23:36.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014149+0000", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056082077000000001 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014197+0000", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7941000000000001e-05 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014247+0000", 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0049000000000001e-05 2024-04-23T21:23:36.877 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40056 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1892352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.861564762, 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056278869000000002, 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.878 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40056 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958081+0000", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958082+0000", 2024-04-23T21:23:36.879 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3300000000000002e-07 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958085+0000", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7329999999999999e-06 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958092+0000", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5820000000000004e-06 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958098+0000", 2024-04-23T21:23:36.880 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3120000000000001e-06 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014272+0000", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056174037000000003 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014312+0000", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0130999999999999e-05 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014360+0000", 2024-04-23T21:23:36.881 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8541000000000002e-05 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40057 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1914880~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.86154099300001, 2024-04-23T21:23:36.882 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056368477, 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40057 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.883 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958105+0000", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6699999999999999e-07 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958108+0000", 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0339999999999999e-06 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.884 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958114+0000", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3990000000000001e-06 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958119+0000", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1699999999999996e-06 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014384+0000", 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056265009999999997 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.885 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014425+0000", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1069000000000003e-05 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014474+0000", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8427999999999998e-05 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40059 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1937408~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:36.886 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.86151998699999, 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056459286999999997, 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40059 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958126+0000", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958127+0000", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958128+0000", 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.849e-06 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.888 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958135+0000", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1619999999999996e-06 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958139+0000", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7190000000000001e-06 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014499+0000", 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056359686999999999 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.889 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014538+0000", 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8720000000000002e-05 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014585+0000", 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7778000000000002e-05 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40061 3.1b 3:dfc72fc2:::rbd_data.111825873f8b.0000000000000003:12 [read 1959936~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.890 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.861500382, 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056551522999999999, 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40061 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958146+0000", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6199999999999999e-07 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958148+0000", 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.069e-06 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.892 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958154+0000", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0870000000000003e-06 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.958159+0000", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5499999999999996e-06 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014610+0000", 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056451016 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.893 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014649+0000", 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9236000000000002e-05 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.014697+0000", 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8202999999999999e-05 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.894 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40421 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 1955840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.845498233, 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056250251000000001, 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40421 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:36.895 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974148+0000", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4700000000000002e-07 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974150+0000", 2024-04-23T21:23:36.896 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0980000000000001e-06 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974156+0000", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7960000000000001e-06 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974161+0000", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4390000000000002e-06 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030249+0000", 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056088300000000001 2024-04-23T21:23:36.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030318+0000", 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9280000000000001e-05 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030398+0000", 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9991e-05 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.898 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40422 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2846720~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.84547972199999, 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056501784999999999, 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40422 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.899 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974166+0000", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974167+0000", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.51e-07 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974169+0000", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3329999999999999e-06 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974176+0000", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3400000000000003e-06 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974180+0000", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7310000000000001e-06 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.901 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030443+0000", 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056262543999999998 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030541+0000", 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7868999999999994e-05 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030668+0000", 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000127417 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.902 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:40424 3.3 3:c41c2cf0:::rbd_data.111825873f8b.0000000000000000:12 [read 2869248~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42)", 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 109.845459277, 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.056686251999999999, 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 40424 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.903 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.974187+0000", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975954+0000", 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017671539999999999 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.904 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975959+0000", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9740000000000001e-06 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975969+0000", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0495e-05 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:46.975977+0000", 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1570000000000003e-06 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.905 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030687+0000", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.054710596 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030782+0000", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.4912000000000002e-05 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:47.030873+0000", 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0964000000000005e-05 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.906 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:53558 3.1a 3:5ac1f2a3:::rbd_object_map.1118a00154cc:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 104.866382065, 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0052677880000001, 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "reached pg", 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 53558 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.907 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953264+0000", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953265+0000", 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.282e-06 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.908 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953271+0000", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.2140000000000004e-06 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953279+0000", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.4980000000000006e-06 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953286+0000", 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2450000000000001e-06 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.909 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953341+0000", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4348999999999998e-05 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953449+0000", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00010873400000000001 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 1", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.953769+0000", 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000319661 2024-04-23T21:23:36.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954704+0000", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000935402 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.954718+0000", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.3062e-05 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958438+0000", 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.003720645 2024-04-23T21:23:36.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:52.958532+0000", 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.3696000000000006e-05 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:36.912 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:36.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-04-23T21:23:37.016 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.016 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:37.016 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:37.016 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.024 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-04-23T21:23:37.126 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.126 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:37.126 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:37.126 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:37.127 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.135 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-04-23T21:23:37.239 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50989 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1392640~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585441272, 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026269324, 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.240 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50989 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654273+0000", 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.241 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654274+0000", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.58e-07 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654277+0000", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.7810000000000001e-06 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654283+0000", 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6100000000000002e-06 2024-04-23T21:23:37.242 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654288+0000", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.921e-06 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680161+0000", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.025872834000000001 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680235+0000", 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4116999999999995e-05 2024-04-23T21:23:37.243 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680543+0000", 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030760299999999999 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.244 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50990 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1572864~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585420148, 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026413491000000001, 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50990 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.245 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654295+0000", 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4900000000000001e-07 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.246 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654297+0000", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0150000000000002e-06 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654303+0000", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9109999999999998e-06 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654313+0000", 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7920000000000004e-06 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.247 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680579+0000", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026266026000000001 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680638+0000", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9799000000000001e-05 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680708+0000", 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.9598999999999996e-05 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.248 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50991 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1949696~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585390881, 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026616763000000002, 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50991 2024-04-23T21:23:37.249 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.250 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654324+0000", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2799999999999996e-07 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654327+0000", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4430000000000002e-06 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654333+0000", 2024-04-23T21:23:37.251 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4309999999999999e-06 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654338+0000", 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5700000000000003e-06 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680737+0000", 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.02639884 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.252 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680827+0000", 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.0142999999999994e-05 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680941+0000", 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000113708 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50992 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2129920~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585370119, 2024-04-23T21:23:37.253 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026775874000000002, 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50992 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.254 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654345+0000", 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1600000000000002e-07 2024-04-23T21:23:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654347+0000", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3429999999999998e-06 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654353+0000", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0769999999999996e-06 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654358+0000", 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5920000000000002e-06 2024-04-23T21:23:37.256 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.680979+0000", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026621293000000001 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681046+0000", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6278999999999993e-05 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681120+0000", 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4874000000000002e-05 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.257 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50993 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2310144~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585351093, 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027012733000000001, 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50993 2024-04-23T21:23:37.258 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654364+0000", 2024-04-23T21:23:37.259 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.0899999999999997e-07 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654366+0000", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3190000000000002e-06 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654372+0000", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8449999999999999e-06 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654376+0000", 2024-04-23T21:23:37.260 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.34e-06 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681162+0000", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.026786071000000002 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681265+0000", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000102082 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681376+0000", 2024-04-23T21:23:37.261 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000111767 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50994 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2490368~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.58533225399999, 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027134688000000001, 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.262 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50994 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654382+0000", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.263 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654383+0000", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6300000000000001e-07 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654395+0000", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2140000000000001e-05 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654402+0000", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6059999999999999e-06 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.264 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654407+0000", 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9949999999999996e-06 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681414+0000", 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027007067999999999 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681459+0000", 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5300000000000003e-05 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.265 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681517+0000", 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8216e-05 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:50995 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2867200~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.58529885199999, 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027259215999999999, 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.266 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 50995 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.267 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654416+0000", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9900000000000001e-07 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654419+0000", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3980000000000002e-06 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654425+0000", 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2439999999999998e-06 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.268 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654429+0000", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.408e-06 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681550+0000", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027120868999999999 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681595+0000", 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4700999999999998e-05 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.269 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681675+0000", 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0097000000000002e-05 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51047 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 311296~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585279106, 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027448061999999999, 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.270 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51047 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:37.271 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654436+0000", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.72e-07 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654438+0000", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.159e-06 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654444+0000", 2024-04-23T21:23:37.272 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6999999999999996e-06 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654448+0000", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1640000000000003e-06 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681698+0000", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027250116000000001 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681780+0000", 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1718999999999995e-05 2024-04-23T21:23:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681884+0000", 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103832 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51048 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 491520~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585261003, 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027676334, 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.274 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51048 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.275 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654454+0000", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0600000000000001e-07 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654456+0000", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.193e-06 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654462+0000", 2024-04-23T21:23:37.276 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1500000000000004e-06 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654467+0000", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4889999999999997e-06 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.681934+0000", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027466655 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682026+0000", 2024-04-23T21:23:37.277 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2545000000000002e-05 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682130+0000", 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000103896 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51049 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 671744~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585241802, 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027799655, 2024-04-23T21:23:37.278 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51049 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.279 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654473+0000", 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1600000000000002e-07 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654481+0000", 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.6240000000000002e-06 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.280 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654487+0000", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6109999999999999e-06 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654492+0000", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5979999999999999e-06 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682159+0000", 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027667383 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682207+0000", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7209999999999997e-05 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682273+0000", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5913000000000004e-05 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51050 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 851968~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:37.282 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585216403, 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028144855, 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51050 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.283 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654498+0000", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654499+0000", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9799999999999999e-07 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654501+0000", 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.187e-06 2024-04-23T21:23:37.284 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654507+0000", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.0700000000000003e-06 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654517+0000", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7699999999999996e-06 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682300+0000", 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.027783351000000001 2024-04-23T21:23:37.285 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682359+0000", 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8533999999999999e-05 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682643+0000", 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00028454499999999998 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.286 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51051 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1228800~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585191278, 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028326107, 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51051 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:37.287 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654523+0000", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654524+0000", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2500000000000001e-07 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654531+0000", 2024-04-23T21:23:37.288 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.888e-06 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654537+0000", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4489999999999996e-06 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654542+0000", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7339999999999999e-06 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682672+0000", 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028130044999999999 2024-04-23T21:23:37.289 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682751+0000", 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.9243e-05 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682850+0000", 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8323e-05 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.290 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51052 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1409024~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.58516626700001, 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028502605, 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51052 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.291 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654548+0000", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654549+0000", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9499999999999998e-07 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654551+0000", 2024-04-23T21:23:37.292 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0930000000000001e-06 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654557+0000", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8409999999999996e-06 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654561+0000", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.279e-06 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682867+0000", 2024-04-23T21:23:37.293 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028306133000000001 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.682950+0000", 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.3066999999999996e-05 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683051+0000", 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000100797 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51053 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1589248~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585142698, 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028683058000000001, 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51053 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654572+0000", 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9999999999999998e-07 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.296 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654574+0000", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.0439999999999998e-06 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654583+0000", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5369999999999997e-06 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654588+0000", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5809999999999998e-06 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.297 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683081+0000", 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028493472999999998 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683164+0000", 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.2681000000000003e-05 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683255+0000", 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.1341999999999994e-05 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.298 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51054 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 1769472~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585119543, 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028868391, 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51054 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.299 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654595+0000", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654596+0000", 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.6800000000000001e-07 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.300 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654597+0000", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.886e-06 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654603+0000", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.9490000000000001e-06 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654607+0000", 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0929999999999996e-06 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.301 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683286+0000", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028678293000000001 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683364+0000", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8152000000000007e-05 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683464+0000", 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.9649999999999996e-05 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.302 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51055 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2146304~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.5850988, 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029059117999999998, 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51055 2024-04-23T21:23:37.303 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654616+0000", 2024-04-23T21:23:37.304 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2e-07 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654618+0000", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.99e-06 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654624+0000", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.84e-06 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654629+0000", 2024-04-23T21:23:37.305 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5800000000000002e-06 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683495+0000", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.028865948999999998 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683576+0000", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1689999999999999e-05 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683675+0000", 2024-04-23T21:23:37.306 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.8648999999999997e-05 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51056 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2326528~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585080499, 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029358321, 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51056 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654634+0000", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.308 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654635+0000", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.3099999999999999e-07 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654636+0000", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.7799999999999999e-06 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654642+0000", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6710000000000004e-06 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.309 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654646+0000", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1099999999999996e-06 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683720+0000", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029073502000000001 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683832+0000", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011282599999999999 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.310 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.683993+0000", 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000160101 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51057 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2506752~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.58506291899999, 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029567067999999998, 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.311 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51057 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.312 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654652+0000", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.7099999999999997e-07 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654658+0000", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3439999999999997e-06 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654665+0000", 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.054e-06 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.313 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654675+0000", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0190999999999999e-05 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684031+0000", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029356594999999999 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684112+0000", 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.0457999999999994e-05 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.314 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684219+0000", 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000107055 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:51058 3.a 3:53653508:::rbd_data.11183e0a89c0.0000000000000000:16 [read 2686976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 105.585033396, 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029740305000000002, 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.315 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 51058 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654681+0000", 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.316 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654682+0000", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3700000000000001e-07 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654684+0000", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2699999999999999e-06 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654690+0000", 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.02e-06 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.317 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.654694+0000", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3830000000000002e-06 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684249+0000", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.029554459000000002 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684326+0000", 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7403000000000004e-05 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.318 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:21:51.684422+0000", 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.5333000000000004e-05 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:61150 3.a 3:53abe656:::rbd_data.1118e08f6c14.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,truncate 6238710] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 96.845778331999995, 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.034315617, 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.319 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 61150 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393936+0000", 2024-04-23T21:23:37.320 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.393939+0000", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3379999999999999e-06 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412382+0000", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.018443588 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412429+0000", 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6616000000000002e-05 2024-04-23T21:23:37.321 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412448+0000", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8841000000000001e-05 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412552+0000", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000104188 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.412723+0000", 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000171256 2024-04-23T21:23:37.322 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.413362+0000", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00063871499999999999 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.424886+0000", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.011523818 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428158+0000", 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0032725110000000001 2024-04-23T21:23:37.323 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428211+0000", 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3054999999999999e-05 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:00.428252+0000", 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0691000000000001e-05 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.324 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.325 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-04-23T21:23:37.326 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.1 primary-affinity to 0.792847 (8519602) 2024-04-23T21:23:37.327 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-04-23T21:23:37.430 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.430 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:37.430 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:37.430 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-04-23T21:23:37.541 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.541 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:37.541 INFO:teuthology.orchestra.run.smithi192.stdout: "complaint_time": 30, 2024-04-23T21:23:37.541 INFO:teuthology.orchestra.run.smithi192.stdout: "num_blocked_ops": 0 2024-04-23T21:23:37.541 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-04-23T21:23:37.654 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "size": 20, 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 600, 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [ 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:63696 3.11 3:8d2a27bd:::rbd_data.1118139b0f86.0000000000000004:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 2097152 write_size 2097152,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 92.527164243000001, 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038610562000000001, 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.655 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 63696 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127445+0000", 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.656 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.127446+0000", 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.083e-06 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129217+0000", 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0017708929999999999 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.657 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129229+0000", 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.198e-05 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129235+0000", 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3840000000000002e-06 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.658 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129292+0000", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6885999999999997e-05 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.129439+0000", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000147508 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.142935+0000", 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.013495534 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.659 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "op_commit", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165887+0000", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.022952237 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.165974+0000", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7268999999999994e-05 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "commit_sent", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166021+0000", 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6363999999999999e-05 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.660 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:05.166055+0000", 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.4424000000000001e-05 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83201 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 622592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.727167840000007, 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038692252000000003, 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.661 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83201 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.662 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.663 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927441+0000", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.46e-07 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927444+0000", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.097e-06 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927449+0000", 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.6280000000000002e-06 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.664 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927454+0000", 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4989999999999996e-06 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966048+0000", 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594779000000003 2024-04-23T21:23:37.665 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966093+0000", 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4697999999999998e-05 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966133+0000", 2024-04-23T21:23:37.666 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0204999999999999e-05 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83215 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.727141320000001, 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038852820000000003, 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.667 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83215 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.668 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927468+0000", 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9000000000000002e-07 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927470+0000", 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2330000000000001e-06 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927476+0000", 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.246e-06 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.669 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927481+0000", 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7219999999999999e-06 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966167+0000", 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038686238999999997 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.670 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966239+0000", 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1227000000000001e-05 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966320+0000", 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.1762999999999996e-05 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83216 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 630784~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:37.671 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.727121256000004, 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038961111999999999, 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83216 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.672 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.673 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927488+0000", 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927669+0000", 2024-04-23T21:23:37.674 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00018091400000000001 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927685+0000", 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6138e-05 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.675 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927695+0000", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0530000000000001e-05 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927702+0000", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4690000000000002e-06 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966348+0000", 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038646013 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.676 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966394+0000", 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6545000000000002e-05 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966449+0000", 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.4503000000000003e-05 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83225 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1056768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.677 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726828585999996, 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038839578, 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83225 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.678 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927780+0000", 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.679 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927781+0000", 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2399999999999999e-07 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927782+0000", 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.2079999999999999e-06 2024-04-23T21:23:37.680 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927785+0000", 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.8770000000000002e-06 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.681 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927787+0000", 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.582e-06 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.682 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966483+0000", 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038696000000000001 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966555+0000", 2024-04-23T21:23:37.683 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2015999999999996e-05 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966620+0000", 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4671000000000005e-05 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.684 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83226 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1114112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726818680999997, 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038942471999999999, 2024-04-23T21:23:37.685 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83226 2024-04-23T21:23:37.686 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.927790+0000", 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.688 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928033+0000", 2024-04-23T21:23:37.689 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000242783 2024-04-23T21:23:37.689 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.689 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.689 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.689 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928037+0000", 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8e-06 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928047+0000", 2024-04-23T21:23:37.690 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.7380000000000006e-06 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928053+0000", 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2589999999999997e-06 2024-04-23T21:23:37.692 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966650+0000", 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038596824000000002 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.693 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.694 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.694 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966689+0000", 2024-04-23T21:23:37.694 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8946999999999998e-05 2024-04-23T21:23:37.694 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.694 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966733+0000", 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.4121e-05 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83228 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1171456~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726515900999999, 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038814434000000002, 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83228 2024-04-23T21:23:37.697 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.698 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928093+0000", 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.699 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928094+0000", 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7000000000000005e-07 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928097+0000", 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.1089999999999998e-06 2024-04-23T21:23:37.700 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928104+0000", 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.5519999999999999e-06 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.701 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928121+0000", 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.6949e-05 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966768+0000", 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038647287000000002 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.702 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966841+0000", 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.2799000000000006e-05 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966907+0000", 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.6167999999999994e-05 2024-04-23T21:23:37.703 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83231 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1097728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:37.704 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726479221000005, 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038901904000000001, 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.705 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83231 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.706 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928130+0000", 2024-04-23T21:23:37.707 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2399999999999998e-07 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928132+0000", 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1160000000000002e-06 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.708 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928139+0000", 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.917e-06 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928144+0000", 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0900000000000004e-06 2024-04-23T21:23:37.709 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966948+0000", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038803749999999998 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.966991+0000", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2985000000000003e-05 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967032+0000", 2024-04-23T21:23:37.710 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0422e-05 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83232 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1155072~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726457597999996, 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039080114999999999, 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.711 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83232 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:37.712 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928151+0000", 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928152+0000", 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8100000000000003e-07 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.713 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928154+0000", 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8980000000000001e-06 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928160+0000", 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.516e-06 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.714 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928165+0000", 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7990000000000001e-06 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967071+0000", 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038905939 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.715 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967157+0000", 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5963000000000003e-05 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967231+0000", 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4518999999999996e-05 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83244 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1048576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.716 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726437352999994, 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039180530999999998, 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83244 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.717 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928172+0000", 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928333+0000", 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00016137700000000001 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.718 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928337+0000", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.5889999999999999e-06 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928355+0000", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.8604999999999999e-05 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928362+0000", 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1269999999999997e-06 2024-04-23T21:23:37.719 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967259+0000", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038897052000000001 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967301+0000", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1714000000000001e-05 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967352+0000", 2024-04-23T21:23:37.720 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1066999999999997e-05 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83245 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1105920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.726236036000003, 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.039125201999999998, 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.721 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83245 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928373+0000", 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.722 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928773+0000", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00040047200000000002 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928781+0000", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.7190000000000001e-06 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928790+0000", 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.5339999999999999e-06 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.723 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928796+0000", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7379999999999997e-06 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967391+0000", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038594521 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967439+0000", 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.8316000000000003e-05 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.724 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967498+0000", 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.8901999999999999e-05 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83246 3.f 3:f6ee7657:::rbd_data.1118f9168b6.0000000000000017:29 [read 1163264~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:37.725 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725801364999995, 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038823907999999997, 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83246 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.726 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928808+0000", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7e-07 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928810+0000", 2024-04-23T21:23:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.2749999999999998e-06 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928817+0000", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.4679999999999997e-06 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.928822+0000", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1490000000000002e-06 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967525+0000", 2024-04-23T21:23:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038702663999999998 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967576+0000", 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.1035000000000002e-05 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967631+0000", 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5847000000000002e-05 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.729 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83255 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1589248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725534539999998, 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038671763999999997, 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83255 2024-04-23T21:23:37.730 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929074+0000", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929075+0000", 2024-04-23T21:23:37.731 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7400000000000003e-07 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929077+0000", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.4650000000000001e-06 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929084+0000", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.686e-06 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929089+0000", 2024-04-23T21:23:37.732 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.0869999999999997e-06 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967660+0000", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038570503999999999 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967699+0000", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9397999999999998e-05 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967746+0000", 2024-04-23T21:23:37.733 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7049999999999998e-05 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83256 3.f 3:f09dd588:::rbd_data.1118f9168b6.000000000000000f:29 [read 1646592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725512339000005, 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038758045999999997, 2024-04-23T21:23:37.734 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83256 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.735 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.736 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929097+0000", 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.8299999999999998e-07 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.737 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929099+0000", 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.9190000000000002e-06 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.738 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929105+0000", 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7019999999999998e-06 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929109+0000", 2024-04-23T21:23:37.739 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.6890000000000003e-06 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967773+0000", 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038663573 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.740 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967812+0000", 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9625000000000001e-05 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.741 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967855+0000", 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.2154999999999999e-05 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83260 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1572864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.742 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725492750000001, 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038864214000000001, 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.743 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83260 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.744 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929116+0000", 2024-04-23T21:23:37.745 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929422+0000", 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00030608600000000001 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.746 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929428+0000", 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.3619999999999997e-06 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929436+0000", 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 8.7069999999999998e-06 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.747 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929443+0000", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3409999999999999e-06 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967881+0000", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038438294999999997 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967939+0000", 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.7834999999999998e-05 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.748 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.967980+0000", 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1588000000000002e-05 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83261 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1687552~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725154438000004, 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038643452000000002, 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.749 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83261 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929454+0000", 2024-04-23T21:23:37.750 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929455+0000", 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.7700000000000005e-07 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929457+0000", 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.3199999999999998e-06 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.751 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929463+0000", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.1759999999999998e-06 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929468+0000", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.809e-06 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968012+0000", 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038543774000000003 2024-04-23T21:23:37.752 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968057+0000", 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.5309000000000003e-05 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968098+0000", 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0587000000000003e-05 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83262 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 1630208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.753 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725133855999999, 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038798206000000002, 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83262 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.754 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929475+0000", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.0499999999999999e-07 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.755 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929478+0000", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.1100000000000001e-06 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929483+0000", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.789e-06 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929488+0000", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.9389999999999996e-06 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.756 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968133+0000", 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038644566999999998 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968206+0000", 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.3256999999999993e-05 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968273+0000", 2024-04-23T21:23:37.757 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.7138999999999995e-05 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83264 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 794624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:37.758 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.725114832000003, 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038891162999999999, 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83264 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.759 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929494+0000", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929691+0000", 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00019726699999999999 2024-04-23T21:23:37.760 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929695+0000", 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.1250000000000003e-06 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.761 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929703+0000", 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.8720000000000002e-06 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929710+0000", 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.2990000000000002e-06 2024-04-23T21:23:37.762 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968302+0000", 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038592070999999999 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968346+0000", 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 4.3956999999999999e-05 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.763 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968385+0000", 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 3.9572e-05 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:83265 3.f 3:f26279dd:::rbd_data.1118f9168b6.0000000000000011:29 [read 851968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e74)", 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 78.724888668000006, 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.038836791000000002, 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.764 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 83265 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.929720+0000", 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.765 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930449+0000", 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.000728653 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930454+0000", 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.023e-06 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.766 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930479+0000", 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 2.5201000000000001e-05 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.930487+0000", 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4880000000000001e-06 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968419+0000", 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.037932780999999999 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.767 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968491+0000", 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.1717999999999993e-05 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:22:18.968557+0000", 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.5926999999999994e-05 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.768 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "description": "osd_op(client.4376.0:140851 3.12 3:4c3ea8ab:::rbd_header.11189a87d8f4:head [watch reconnect cookie 140596427927216 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "initiated_at": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "age": 24.311806832999999, 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0289291229999999, 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "continuous": false, 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "type_data": { 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "flag_point": "started", 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client_info": { 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client": "client.4376", 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "tid": 140851 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "events": [ 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.769 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "initiated", 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "header_read", 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342802+0000", 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "throttled", 2024-04-23T21:23:37.770 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342804+0000", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.576e-06 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "all_read", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342809+0000", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 5.5969999999999999e-06 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "dispatched", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342821+0000", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1286e-05 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "queued_for_pg", 2024-04-23T21:23:37.771 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342830+0000", 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 9.2560000000000008e-06 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342904+0000", 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 7.4219000000000002e-05 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for peered", 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:13.342915+0000", 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.1090999999999999e-05 2024-04-23T21:23:37.772 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364226+0000", 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 1.0213109119999999 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "waiting for rw locks", 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.364289+0000", 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 6.3180000000000002e-05 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.773 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "reached_pg", 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366791+0000", 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0025013290000000001 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "started", 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.366909+0000", 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.00011882799999999999 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: { 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "event": "done", 2024-04-23T21:23:37.774 INFO:teuthology.orchestra.run.smithi192.stdout: "time": "2024-04-23T21:23:14.371731+0000", 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: "duration": 0.0048218489999999996 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-23T21:23:37.775 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:37.776 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-04-23T21:23:37.879 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:37.880 INFO:teuthology.orchestra.run.smithi111.stdout: "num_ops": 0 2024-04-23T21:23:37.880 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:37.888 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-04-23T21:23:37.989 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:37.990 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [], 2024-04-23T21:23:37.990 INFO:teuthology.orchestra.run.smithi111.stdout: "complaint_time": 30, 2024-04-23T21:23:37.990 INFO:teuthology.orchestra.run.smithi111.stdout: "num_blocked_ops": 0 2024-04-23T21:23:37.990 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:37.999 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-04-23T21:23:38.101 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "size": 20, 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 600, 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "ops": [ 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:48099 3.19 3.cf0e4319 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 109.160822379, 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.994712865, 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.102 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "reached pg", 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 48099 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.103 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940785+0000", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940786+0000", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3519999999999999e-06 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940792+0000", 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.524e-06 2024-04-23T21:23:38.104 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940802+0000", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9969999999999993e-06 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:48.940812+0000", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0496000000000001e-05 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235610+0000", 2024-04-23T21:23:38.105 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.29479781500000002 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.235620+0000", 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0411000000000001e-05 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.106 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935485+0000", 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.69986423799999997 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:21:49.935498+0000", 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3032e-05 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70925 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3276800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.107 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331423450000003, 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087686577000000002, 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70925 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:38.108 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.109 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770184+0000", 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1399999999999997e-07 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770186+0000", 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2809999999999998e-06 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770192+0000", 2024-04-23T21:23:38.110 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.801e-06 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770200+0000", 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9980000000000003e-06 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857683+0000", 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087482796000000002 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.111 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857770+0000", 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6613999999999994e-05 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857870+0000", 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100673 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70926 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 3932160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:38.112 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331400516000002, 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087783575000000003, 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70926 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.113 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770207+0000", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.77e-07 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770214+0000", 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.708e-06 2024-04-23T21:23:38.114 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770221+0000", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1230000000000002e-06 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770225+0000", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4329999999999997e-06 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857906+0000", 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087680835999999998 2024-04-23T21:23:38.115 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857944+0000", 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.7969999999999997e-05 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.116 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.857990+0000", 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6128000000000003e-05 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.117 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70927 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 4587520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331375197, 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087981955000000001, 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70927 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.118 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.119 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.119 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:38.119 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.119 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.120 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.120 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770232+0000", 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9099999999999999e-07 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.121 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770235+0000", 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.069e-06 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770240+0000", 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.9290000000000003e-06 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.122 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770245+0000", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2119999999999997e-06 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858026+0000", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087781673000000005 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858113+0000", 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.6866999999999993e-05 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.123 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858214+0000", 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000100814 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70928 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5242880~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.124 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331356478000004, 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088070949999999995, 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70928 2024-04-23T21:23:38.125 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.126 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770251+0000", 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.03e-07 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770254+0000", 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.3460000000000001e-06 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.127 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770262+0000", 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.3319999999999992e-06 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770271+0000", 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8060000000000001e-06 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.128 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858229+0000", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.087958587000000005 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858275+0000", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6134000000000002e-05 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858322+0000", 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6341999999999998e-05 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.129 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70929 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 5898240~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331313797000007, 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088257865000000005, 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.130 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70929 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.131 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770293+0000", 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770294+0000", 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.2099999999999996e-07 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.132 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770297+0000", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.9660000000000001e-06 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770304+0000", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.6379999999999998e-06 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770313+0000", 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.9639999999999992e-06 2024-04-23T21:23:38.133 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858358+0000", 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088044872999999996 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.134 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858449+0000", 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.1779000000000003e-05 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858551+0000", 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000102024 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.135 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70930 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7208960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331287657000004, 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088339306000000006, 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70930 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.136 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770320+0000", 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3099999999999997e-07 2024-04-23T21:23:38.137 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770322+0000", 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.21e-06 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770328+0000", 2024-04-23T21:23:38.138 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.3800000000000002e-06 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770333+0000", 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6979999999999997e-06 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858566+0000", 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088233076999999993 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.139 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858610+0000", 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4891000000000001e-05 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858659+0000", 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8418999999999998e-05 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70931 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 7864320~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331269739000007, 2024-04-23T21:23:38.140 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088543169000000005, 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70931 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.141 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770338+0000", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.0600000000000001e-07 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770352+0000", 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.3963e-05 2024-04-23T21:23:38.142 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770358+0000", 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.3130000000000004e-06 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770362+0000", 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2849999999999996e-06 2024-04-23T21:23:38.143 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858695+0000", 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088332758999999997 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858780+0000", 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4571000000000001e-05 2024-04-23T21:23:38.144 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858881+0000", 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00010097199999999999 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.145 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70932 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 8519680~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.33123836, 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088620862999999994, 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70932 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.146 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770369+0000", 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0699999999999998e-07 2024-04-23T21:23:38.147 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770371+0000", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.809e-06 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770376+0000", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9969999999999998e-06 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770380+0000", 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8550000000000004e-06 2024-04-23T21:23:38.148 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858896+0000", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088516154 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858942+0000", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6372000000000003e-05 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.858990+0000", 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7268999999999998e-05 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.149 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70933 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9175040~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331221231000001, 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088826148999999993, 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.150 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70933 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.151 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770386+0000", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.84e-07 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770388+0000", 2024-04-23T21:23:38.152 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6360000000000001e-06 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770393+0000", 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2689999999999999e-06 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.153 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770402+0000", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.8710000000000003e-06 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859026+0000", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088623489999999999 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859110+0000", 2024-04-23T21:23:38.154 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4612999999999998e-05 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859212+0000", 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000101986 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70934 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 9830400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:38.155 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331192693000006, 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088906576000000001, 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70934 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.156 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770415+0000", 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.5499999999999998e-07 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.157 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770417+0000", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.9250000000000002e-06 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770426+0000", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.3689999999999996e-06 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770430+0000", 2024-04-23T21:23:38.158 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.0949999999999998e-06 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859228+0000", 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088797424999999999 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859275+0000", 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6638000000000003e-05 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.159 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859321+0000", 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6569000000000001e-05 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70935 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 10485760~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331149707999998, 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089093640000000002, 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.160 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70935 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.161 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770458+0000", 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.2599999999999998e-07 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770460+0000", 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.469e-06 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.162 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770467+0000", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.1319999999999999e-06 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770472+0000", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2480000000000004e-06 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859357+0000", 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.088885325000000001 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.163 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859452+0000", 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.4973e-05 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859551+0000", 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.9067000000000005e-05 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.164 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70936 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11141120~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331122471, 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089188365000000006, 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.165 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70936 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.166 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770485+0000", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.7599999999999997e-07 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770488+0000", 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.5330000000000001e-06 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.167 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770494+0000", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.8719999999999999e-06 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770502+0000", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4870000000000002e-06 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859585+0000", 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089082475999999994 2024-04-23T21:23:38.168 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859624+0000", 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.9561000000000003e-05 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859673+0000", 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.8959999999999999e-05 2024-04-23T21:23:38.169 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70937 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 11796480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331098683999997, 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089370131000000005, 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.170 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70937 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.171 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770509+0000", 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.4199999999999996e-07 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.172 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770511+0000", 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2280000000000001e-06 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770517+0000", 2024-04-23T21:23:38.173 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2329999999999997e-06 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770524+0000", 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.4279999999999999e-06 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.174 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859709+0000", 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089185153000000003 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.175 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859788+0000", 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9342e-05 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859879+0000", 2024-04-23T21:23:38.176 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.0204999999999994e-05 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70938 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13107200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331073442000005, 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089451066999999995, 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.178 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70938 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.179 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.180 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.181 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770534+0000", 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.1800000000000001e-07 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.182 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770536+0000", 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.765e-06 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770542+0000", 2024-04-23T21:23:38.183 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.6969999999999998e-06 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770546+0000", 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6360000000000002e-06 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.184 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859892+0000", 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089345919999999995 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.185 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859939+0000", 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6421000000000002e-05 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.186 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.859985+0000", 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.621e-05 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70939 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 13762560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.187 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.331054675999994, 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089644018000000006, 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.188 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70939 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.189 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770553+0000", 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.2000000000000001e-07 2024-04-23T21:23:38.190 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.191 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.191 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.191 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770554+0000", 2024-04-23T21:23:38.191 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5260000000000001e-06 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770565+0000", 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0404999999999999e-05 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770576+0000", 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0781999999999999e-05 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.192 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860019+0000", 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089443028999999993 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860099+0000", 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.9958999999999998e-05 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860197+0000", 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 9.7997000000000004e-05 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.193 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70940 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 14417920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.330997715999999, 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089706706999999997, 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.194 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70940 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.195 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770610+0000", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.9100000000000004e-07 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770612+0000", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.1689999999999999e-06 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.196 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770621+0000", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4780000000000008e-06 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770625+0000", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.5170000000000001e-06 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860232+0000", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089606893000000007 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.197 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860270+0000", 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8000000000000002e-05 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860316+0000", 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6159000000000003e-05 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70941 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15073280~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:38.198 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.330975981999998, 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089908768, 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70941 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.199 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770631+0000", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770632+0000", 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.4700000000000002e-07 2024-04-23T21:23:38.200 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770634+0000", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.95e-06 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770642+0000", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.4239999999999993e-06 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770647+0000", 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6149999999999999e-06 2024-04-23T21:23:38.201 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860349+0000", 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.089702636000000002 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860429+0000", 2024-04-23T21:23:38.202 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 8.0147000000000004e-05 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860540+0000", 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.000110549 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:70942 3.4 3:275f11a6:::rbd_data.11189d22003d.0000000000000003:25 [sparse-read 15728640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 84.330951760999994, 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.090005835000000006, 2024-04-23T21:23:38.203 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "started", 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 70942 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.204 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770655+0000", 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770656+0000", 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 5.2900000000000004e-07 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.205 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.770662+0000", 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.0970000000000002e-06 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836889+0000", 2024-04-23T21:23:38.206 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.066226585000000004 2024-04-23T21:23:38.207 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.207 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.207 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.207 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.836905+0000", 2024-04-23T21:23:38.207 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5908000000000001e-05 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860576+0000", 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.023671385999999999 2024-04-23T21:23:38.208 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860614+0000", 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8449999999999999e-05 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.209 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:13.860661+0000", 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 4.6879999999999998e-05 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.210 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "description": "osd_op(client.4376.0:131746 3.19 3:9cc861ac:::rbd_object_map.1118ba99ca9c:head [call lock.assert_locked in=17b,call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "initiated_at": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "age": 64.901682706000003, 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98849368999999998, 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "continuous": false, 2024-04-23T21:23:38.211 INFO:teuthology.orchestra.run.smithi111.stdout: "type_data": { 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_info": { 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client": "client.4376", 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: "client_addr": "172.21.15.111:0/1125244421", 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: "tid": 131746 2024-04-23T21:23:38.212 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: "events": [ 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "initiated", 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.213 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "header_read", 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199925+0000", 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.214 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "throttled", 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199926+0000", 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.5009999999999999e-06 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "all_read", 2024-04-23T21:23:38.215 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199933+0000", 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 7.1180000000000003e-06 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "dispatched", 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199949+0000", 2024-04-23T21:23:38.216 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.6079999999999999e-05 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "queued_for_pg", 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.199960+0000", 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.0533000000000001e-05 2024-04-23T21:23:38.217 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200029+0000", 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 6.9620000000000001e-05 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.218 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for peered", 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:33.200041+0000", 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 1.1471000000000001e-05 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.219 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "reached_pg", 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186586+0000", 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.98654473099999995 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "started", 2024-04-23T21:23:38.220 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.186736+0000", 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00015043500000000001 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "waiting for subops from 4", 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187147+0000", 2024-04-23T21:23:38.221 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00041065999999999997 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "op_commit", 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.187929+0000", 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00078195000000000001 2024-04-23T21:23:38.222 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "sub_op_commit_rec", 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188358+0000", 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 0.00042912700000000001 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.223 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "commit_sent", 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188396+0000", 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 3.8056999999999998e-05 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: { 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: "event": "done", 2024-04-23T21:23:38.224 INFO:teuthology.orchestra.run.smithi111.stdout: "time": "2024-04-23T21:22:34.188418+0000", 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: "duration": 2.2407000000000001e-05 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-23T21:23:38.225 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-23T21:23:38.226 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:23:38.255 INFO:teuthology.orchestra.run.smithi111.stderr:noscrub is unset 2024-04-23T21:23:38.271 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-04-23T21:23:39.266 INFO:teuthology.orchestra.run.smithi111.stderr:nodeep-scrub is unset 2024-04-23T21:23:42.329 INFO:tasks.thrashosds.thrasher:reviving osd 2024-04-23T21:23:42.329 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-04-23T21:23:42.329 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-23T21:23:42.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-23T21:23:42.333 INFO:tasks.ceph.osd.5:Started 2024-04-23T21:23:42.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:23:42.442 INFO:teuthology.orchestra.run.smithi192.stderr:no valid command found; 10 closest matches: 2024-04-23T21:23:42.442 INFO:teuthology.orchestra.run.smithi192.stderr:0 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:1 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:2 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:abort 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:assert 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:config diff 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:config diff get 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:config get 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:config help [] 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:config set ... 2024-04-23T21:23:42.443 INFO:teuthology.orchestra.run.smithi192.stderr:admin_socket: invalid command 2024-04-23T21:23:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-23T21:23:42.445 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-04-23T21:23:42.928 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:23:42.928+0000 7fcbdd60c640 -1 Falling back to public interface 2024-04-23T21:23:46.409 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:23:46.409+0000 7fcbdd60c640 -1 osd.5 104 log_to_monitors true 2024-04-23T21:23:47.327 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-04-23T21:23:47.327+0000 7fcbd0d4a640 -1 osd.5 156 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T21:23:47.446 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-04-23T21:23:47.557 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-23T21:23:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "ops": [], 2024-04-23T21:23:47.558 INFO:teuthology.orchestra.run.smithi192.stdout: "num_ops": 0 2024-04-23T21:23:47.558 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-23T21:23:47.567 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T21:23:47.567 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-04-23T21:23:47.567 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-04-23T21:23:47.871 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:23:47.872+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e156 definitely_dead 0 2024-04-23T21:23:48.293 INFO:teuthology.orchestra.run.smithi111.stderr:marked in osd.7. 2024-04-23T21:23:48.312 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-04-23T21:23:48.313 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T21:23:48.313 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-04-23T21:23:48.313 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-04-23T21:23:48.607 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:23:48.608+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e157 definitely_dead 0 2024-04-23T21:23:49.298 INFO:teuthology.orchestra.run.smithi111.stderr:marked in osd.3. 2024-04-23T21:23:49.319 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-04-23T21:23:49.320 INFO:tasks.thrashosds.thrasher:inning osd 2024-04-23T21:23:49.320 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-04-23T21:23:49.320 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-04-23T21:23:49.617 INFO:tasks.ceph.mon.a.smithi111.stderr:2024-04-23T21:23:49.618+0000 7fd87b0eb640 -1 mon.a@0(leader).osd e158 definitely_dead 0 2024-04-23T21:23:50.301 INFO:teuthology.orchestra.run.smithi111.stderr:marked in osd.0. 2024-04-23T21:23:50.326 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-04-23T21:23:50.327 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-23T21:23:50.464 INFO:teuthology.orchestra.run.smithi111.stdout:mon.a: {} 2024-04-23T21:23:50.464 INFO:teuthology.orchestra.run.smithi111.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:23:50.474 INFO:teuthology.orchestra.run.smithi111.stdout:mon.b: {} 2024-04-23T21:23:50.474 INFO:teuthology.orchestra.run.smithi111.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:23:50.483 INFO:teuthology.orchestra.run.smithi111.stdout:mon.c: {} 2024-04-23T21:23:50.483 INFO:teuthology.orchestra.run.smithi111.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-23T21:23:50.495 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-04-23T21:23:51.307 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.0 to 1 (10000) 2024-04-23T21:23:51.331 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-04-23T21:23:52.197 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-23T21:23:52.219 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-04-23T21:23:53.193 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.2 to 1 (10000) 2024-04-23T21:23:53.217 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-04-23T21:23:54.195 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-23T21:23:54.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-04-23T21:23:55.198 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.3 to 1 (10000) 2024-04-23T21:23:55.213 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-04-23T21:23:56.339 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-23T21:23:56.356 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-04-23T21:23:57.193 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.4 to 1 (10000) 2024-04-23T21:23:57.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-04-23T21:23:58.347 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-23T21:23:58.361 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-04-23T21:23:59.345 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.6 to 1 (10000) 2024-04-23T21:23:59.364 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-04-23T21:24:00.353 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-23T21:24:00.369 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-04-23T21:24:01.367 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.7 to 1 (10000) 2024-04-23T21:24:01.386 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-04-23T21:24:02.372 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-23T21:24:02.387 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-04-23T21:24:03.374 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.1 to 1 (10000) 2024-04-23T21:24:03.390 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-04-23T21:24:04.380 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-23T21:24:04.397 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-04-23T21:24:05.400 INFO:teuthology.orchestra.run.smithi111.stderr:reweighted osd.5 to 1 (10000) 2024-04-23T21:24:05.415 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-04-23T21:24:06.409 INFO:teuthology.orchestra.run.smithi111.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-23T21:24:06.425 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-23T21:24:06.426 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-23T21:24:06.426 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-23T21:24:06.426 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-23T21:24:06.426 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-23T21:24:06.426 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:24:06.718 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:06.718 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":175,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:24:06.401539+0000","last_up_change":"2024-04-23T21:23:48.290514+0000","last_in_change":"2024-04-23T21:23:50.294338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T21:21:08.380744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-23T21:21:09.608446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-23T21:21:16.010288+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":51,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":45,"up_thru":173,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6817","nonce":723341542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6819","nonce":723341542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6823","nonce":723341542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6821","nonce":723341542}]},"public_addr":"172.21.15.111:6817/723341542","cluster_addr":"172.21.15.111:6819/723341542","heartbeat_back_addr":"172.21.15.111:6823/723341542","heartbeat_front_addr":"172.21.15.111:6821/723341542","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":104,"up_from":157,"up_thru":160,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6801","nonce":1231536685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6803","nonce":1231536685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6807","nonce":1231536685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6805","nonce":1231536685}]},"public_addr":"172.21.15.192:6801/1231536685","cluster_addr":"172.21.15.192:6803/1231536685","heartbeat_back_addr":"172.21.15.192:6807/1231536685","heartbeat_front_addr":"172.21.15.192:6805/1231536685","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.914033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.903496+0000","dead_epoch":18},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.934930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:05.001485+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.979253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.839146+0000","dead_epoch":105},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.963622+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.946000+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.1f","osds":[2,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:24:06.731 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-23T21:24:06.731 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:24:07.015 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:07.016 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":175,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:24:06.401539+0000","last_up_change":"2024-04-23T21:23:48.290514+0000","last_in_change":"2024-04-23T21:23:50.294338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T21:21:08.380744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-23T21:21:09.608446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-23T21:21:16.010288+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":51,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":45,"up_thru":173,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6817","nonce":723341542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6819","nonce":723341542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6823","nonce":723341542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6821","nonce":723341542}]},"public_addr":"172.21.15.111:6817/723341542","cluster_addr":"172.21.15.111:6819/723341542","heartbeat_back_addr":"172.21.15.111:6823/723341542","heartbeat_front_addr":"172.21.15.111:6821/723341542","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":104,"up_from":157,"up_thru":160,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6801","nonce":1231536685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6803","nonce":1231536685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6807","nonce":1231536685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6805","nonce":1231536685}]},"public_addr":"172.21.15.192:6801/1231536685","cluster_addr":"172.21.15.192:6803/1231536685","heartbeat_back_addr":"172.21.15.192:6807/1231536685","heartbeat_front_addr":"172.21.15.192:6805/1231536685","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.914033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.903496+0000","dead_epoch":18},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.934930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:05.001485+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.979253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.839146+0000","dead_epoch":105},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.963622+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.946000+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.1f","osds":[2,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:24:07.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-23T21:24:07.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-23T21:24:07.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-23T21:24:07.030 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-23T21:24:07.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-23T21:24:07.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-23T21:24:07.031 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-23T21:24:07.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-23T21:24:07.263 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:07.263 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T21:24:07.270 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:07.271 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T21:24:07.272 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:07.272 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T21:24:07.273 INFO:teuthology.orchestra.run.smithi111.stdout:193273528349 2024-04-23T21:24:07.273 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T21:24:07.275 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:07.275 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T21:24:07.276 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:07.276 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T21:24:07.279 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:07.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T21:24:07.280 INFO:teuthology.orchestra.run.smithi111.stdout:674309865477 2024-04-23T21:24:07.280 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T21:24:07.692 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:07.701 INFO:teuthology.orchestra.run.smithi111.stderr:2024-04-23T21:24:07.701+0000 7f546b7fe640 0 --2- 172.21.15.111:0/2428656082 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f546c1464f0 0x7f546c13bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-23T21:24:07.712 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.3 2024-04-23T21:24:07.741 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:07.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.2 2024-04-23T21:24:07.833 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:07.851 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.6 2024-04-23T21:24:07.864 INFO:teuthology.orchestra.run.smithi111.stdout:193273528348 2024-04-23T21:24:07.878 INFO:tasks.ceph.ceph_manager.ceph:need seq 193273528349 got 193273528348 for osd.1 2024-04-23T21:24:07.883 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:07.896 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.7 2024-04-23T21:24:07.899 INFO:teuthology.orchestra.run.smithi111.stdout:674309865476 2024-04-23T21:24:07.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 674309865477 got 674309865476 for osd.5 2024-04-23T21:24:07.940 INFO:teuthology.orchestra.run.smithi111.stdout:55834574885 2024-04-23T21:24:07.953 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574885 for osd.0 2024-04-23T21:24:08.018 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:08.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574886 for osd.4 2024-04-23T21:24:08.713 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-23T21:24:08.755 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-23T21:24:08.852 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-23T21:24:08.879 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-23T21:24:08.897 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-23T21:24:08.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-23T21:24:08.957 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-23T21:24:09.032 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-23T21:24:09.074 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:09.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.3 2024-04-23T21:24:09.097 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.148 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:09.169 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.2 2024-04-23T21:24:09.169 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.281 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:09.295 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.6 2024-04-23T21:24:09.295 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.330 INFO:teuthology.orchestra.run.smithi111.stdout:674309865478 2024-04-23T21:24:09.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 674309865477 got 674309865478 for osd.5 2024-04-23T21:24:09.343 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.418 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:09.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.7 2024-04-23T21:24:09.431 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.438 INFO:teuthology.orchestra.run.smithi111.stdout:193273528349 2024-04-23T21:24:09.451 INFO:tasks.ceph.ceph_manager.ceph:need seq 193273528349 got 193273528349 for osd.1 2024-04-23T21:24:09.451 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.511 INFO:teuthology.orchestra.run.smithi111.stdout:55834574886 2024-04-23T21:24:09.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574886 got 55834574886 for osd.0 2024-04-23T21:24:09.524 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.537 INFO:teuthology.orchestra.run.smithi111.stdout:55834574887 2024-04-23T21:24:09.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.4 2024-04-23T21:24:09.551 DEBUG:teuthology.parallel:result is None 2024-04-23T21:24:09.551 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-04-23T21:24:09.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:09.877 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:09.878 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:09.888 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-23T21:24:09.627366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380372,"kb_used_data":16672,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594348,"statfs":{"total":773094113280,"available":772704612352,"internally_reserved":0,"allocated":17072128,"data_stored":9494914,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244093"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8752,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436735+0000","last_change":"2024-04-23T21:23:49.395000+0000","last_active":"2024-04-23T21:24:02.436735+0000","last_peered":"2024-04-23T21:24:02.436735+0000","last_clean":"2024-04-23T21:24:02.436735+0000","last_became_active":"2024-04-23T21:23:49.313320+0000","last_became_peered":"2024-04-23T21:23:49.313320+0000","last_unstale":"2024-04-23T21:24:02.436735+0000","last_undegraded":"2024-04-23T21:24:02.436735+0000","last_fullsized":"2024-04-23T21:24:02.436735+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:00.953641+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.064094391000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3731,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436764+0000","last_change":"2024-04-23T21:23:49.438136+0000","last_active":"2024-04-23T21:24:02.436764+0000","last_peered":"2024-04-23T21:24:02.436764+0000","last_clean":"2024-04-23T21:24:02.436764+0000","last_became_active":"2024-04-23T21:23:49.313703+0000","last_became_peered":"2024-04-23T21:23:49.313703+0000","last_unstale":"2024-04-23T21:24:02.436764+0000","last_undegraded":"2024-04-23T21:24:02.436764+0000","last_fullsized":"2024-04-23T21:24:02.436764+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:02.636441+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.063117355999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417056+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417056+0000","last_peered":"2024-04-23T21:24:07.417056+0000","last_clean":"2024-04-23T21:24:07.417056+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417056+0000","last_undegraded":"2024-04-23T21:24:07.417056+0000","last_fullsized":"2024-04-23T21:24:07.417056+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:25:04.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3494,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416076+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416076+0000","last_peered":"2024-04-23T21:24:07.416076+0000","last_clean":"2024-04-23T21:24:07.416076+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416076+0000","last_undegraded":"2024-04-23T21:24:07.416076+0000","last_fullsized":"2024-04-23T21:24:07.416076+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:36.032019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5325,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-04-23T21:24:06.409211+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:06.409211+0000","last_peered":"2024-04-23T21:24:06.409211+0000","last_clean":"2024-04-23T21:24:06.409211+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:06.409211+0000","last_undegraded":"2024-04-23T21:24:06.409211+0000","last_fullsized":"2024-04-23T21:24:06.409211+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3109,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436400+0000","last_change":"2024-04-23T21:23:53.295392+0000","last_active":"2024-04-23T21:24:02.436400+0000","last_peered":"2024-04-23T21:24:02.436400+0000","last_clean":"2024-04-23T21:24:02.436400+0000","last_became_active":"2024-04-23T21:23:53.206512+0000","last_became_peered":"2024-04-23T21:23:53.206512+0000","last_unstale":"2024-04-23T21:24:02.436400+0000","last_undegraded":"2024-04-23T21:24:02.436400+0000","last_fullsized":"2024-04-23T21:24:02.436400+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:20:23.215602+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.084884373999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417233+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417233+0000","last_peered":"2024-04-23T21:24:07.417233+0000","last_clean":"2024-04-23T21:24:07.417233+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417233+0000","last_undegraded":"2024-04-23T21:24:07.417233+0000","last_fullsized":"2024-04-23T21:24:07.417233+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:18:30.932913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8777,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416829+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416829+0000","last_peered":"2024-04-23T21:24:07.416829+0000","last_clean":"2024-04-23T21:24:07.416829+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416829+0000","last_undegraded":"2024-04-23T21:24:07.416829+0000","last_fullsized":"2024-04-23T21:24:07.416829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:59.559020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436249+0000","last_change":"2024-04-23T21:23:51.305630+0000","last_active":"2024-04-23T21:24:02.436249+0000","last_peered":"2024-04-23T21:24:02.436249+0000","last_clean":"2024-04-23T21:24:02.436249+0000","last_became_active":"2024-04-23T21:23:51.305331+0000","last_became_peered":"2024-04-23T21:23:51.305331+0000","last_unstale":"2024-04-23T21:24:02.436249+0000","last_undegraded":"2024-04-23T21:24:02.436249+0000","last_fullsized":"2024-04-23T21:24:02.436249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:27:24.622720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2188,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436258+0000","last_change":"2024-04-23T21:23:53.287328+0000","last_active":"2024-04-23T21:24:02.436258+0000","last_peered":"2024-04-23T21:24:02.436258+0000","last_clean":"2024-04-23T21:24:02.436258+0000","last_became_active":"2024-04-23T21:23:53.206462+0000","last_became_peered":"2024-04-23T21:23:53.206462+0000","last_unstale":"2024-04-23T21:24:02.436258+0000","last_undegraded":"2024-04-23T21:24:02.436258+0000","last_fullsized":"2024-04-23T21:24:02.436258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:58.138448+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.079455974999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4155,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416423+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416423+0000","last_peered":"2024-04-23T21:24:07.416423+0000","last_clean":"2024-04-23T21:24:07.416423+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416423+0000","last_undegraded":"2024-04-23T21:24:07.416423+0000","last_fullsized":"2024-04-23T21:24:07.416423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:49.518562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865478,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30940,"kb_used_data":2060,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340900,"statfs":{"total":96636764160,"available":96605081600,"internally_reserved":0,"allocated":2109440,"data_stored":1034886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":2112,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":2162688,"data_stored":1625273,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:09.891 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:10.175 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:10.176 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:10.190 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-23T21:24:09.627366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380372,"kb_used_data":16672,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594348,"statfs":{"total":773094113280,"available":772704612352,"internally_reserved":0,"allocated":17072128,"data_stored":9494914,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244093"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8752,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436735+0000","last_change":"2024-04-23T21:23:49.395000+0000","last_active":"2024-04-23T21:24:02.436735+0000","last_peered":"2024-04-23T21:24:02.436735+0000","last_clean":"2024-04-23T21:24:02.436735+0000","last_became_active":"2024-04-23T21:23:49.313320+0000","last_became_peered":"2024-04-23T21:23:49.313320+0000","last_unstale":"2024-04-23T21:24:02.436735+0000","last_undegraded":"2024-04-23T21:24:02.436735+0000","last_fullsized":"2024-04-23T21:24:02.436735+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:00.953641+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.064094391000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3731,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436764+0000","last_change":"2024-04-23T21:23:49.438136+0000","last_active":"2024-04-23T21:24:02.436764+0000","last_peered":"2024-04-23T21:24:02.436764+0000","last_clean":"2024-04-23T21:24:02.436764+0000","last_became_active":"2024-04-23T21:23:49.313703+0000","last_became_peered":"2024-04-23T21:23:49.313703+0000","last_unstale":"2024-04-23T21:24:02.436764+0000","last_undegraded":"2024-04-23T21:24:02.436764+0000","last_fullsized":"2024-04-23T21:24:02.436764+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:02.636441+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.063117355999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417056+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417056+0000","last_peered":"2024-04-23T21:24:07.417056+0000","last_clean":"2024-04-23T21:24:07.417056+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417056+0000","last_undegraded":"2024-04-23T21:24:07.417056+0000","last_fullsized":"2024-04-23T21:24:07.417056+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:25:04.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3494,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416076+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416076+0000","last_peered":"2024-04-23T21:24:07.416076+0000","last_clean":"2024-04-23T21:24:07.416076+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416076+0000","last_undegraded":"2024-04-23T21:24:07.416076+0000","last_fullsized":"2024-04-23T21:24:07.416076+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:36.032019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5325,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-04-23T21:24:06.409211+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:06.409211+0000","last_peered":"2024-04-23T21:24:06.409211+0000","last_clean":"2024-04-23T21:24:06.409211+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:06.409211+0000","last_undegraded":"2024-04-23T21:24:06.409211+0000","last_fullsized":"2024-04-23T21:24:06.409211+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3109,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436400+0000","last_change":"2024-04-23T21:23:53.295392+0000","last_active":"2024-04-23T21:24:02.436400+0000","last_peered":"2024-04-23T21:24:02.436400+0000","last_clean":"2024-04-23T21:24:02.436400+0000","last_became_active":"2024-04-23T21:23:53.206512+0000","last_became_peered":"2024-04-23T21:23:53.206512+0000","last_unstale":"2024-04-23T21:24:02.436400+0000","last_undegraded":"2024-04-23T21:24:02.436400+0000","last_fullsized":"2024-04-23T21:24:02.436400+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:20:23.215602+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.084884373999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417233+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417233+0000","last_peered":"2024-04-23T21:24:07.417233+0000","last_clean":"2024-04-23T21:24:07.417233+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417233+0000","last_undegraded":"2024-04-23T21:24:07.417233+0000","last_fullsized":"2024-04-23T21:24:07.417233+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:18:30.932913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8777,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416829+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416829+0000","last_peered":"2024-04-23T21:24:07.416829+0000","last_clean":"2024-04-23T21:24:07.416829+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416829+0000","last_undegraded":"2024-04-23T21:24:07.416829+0000","last_fullsized":"2024-04-23T21:24:07.416829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:59.559020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436249+0000","last_change":"2024-04-23T21:23:51.305630+0000","last_active":"2024-04-23T21:24:02.436249+0000","last_peered":"2024-04-23T21:24:02.436249+0000","last_clean":"2024-04-23T21:24:02.436249+0000","last_became_active":"2024-04-23T21:23:51.305331+0000","last_became_peered":"2024-04-23T21:23:51.305331+0000","last_unstale":"2024-04-23T21:24:02.436249+0000","last_undegraded":"2024-04-23T21:24:02.436249+0000","last_fullsized":"2024-04-23T21:24:02.436249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:27:24.622720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2188,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436258+0000","last_change":"2024-04-23T21:23:53.287328+0000","last_active":"2024-04-23T21:24:02.436258+0000","last_peered":"2024-04-23T21:24:02.436258+0000","last_clean":"2024-04-23T21:24:02.436258+0000","last_became_active":"2024-04-23T21:23:53.206462+0000","last_became_peered":"2024-04-23T21:23:53.206462+0000","last_unstale":"2024-04-23T21:24:02.436258+0000","last_undegraded":"2024-04-23T21:24:02.436258+0000","last_fullsized":"2024-04-23T21:24:02.436258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:58.138448+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.079455974999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4155,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416423+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416423+0000","last_peered":"2024-04-23T21:24:07.416423+0000","last_clean":"2024-04-23T21:24:07.416423+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416423+0000","last_undegraded":"2024-04-23T21:24:07.416423+0000","last_fullsized":"2024-04-23T21:24:07.416423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:49.518562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865478,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30940,"kb_used_data":2060,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340900,"statfs":{"total":96636764160,"available":96605081600,"internally_reserved":0,"allocated":2109440,"data_stored":1034886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":2112,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":2162688,"data_stored":1625273,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:10.192 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-04-23T21:24:10.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-23T21:24:10.495 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-23T21:24:10.504 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-23T21:24:10.505 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:10.790 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:10.791 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:10.803 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-23T21:24:09.627366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380372,"kb_used_data":16672,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594348,"statfs":{"total":773094113280,"available":772704612352,"internally_reserved":0,"allocated":17072128,"data_stored":9494914,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244093"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8752,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436735+0000","last_change":"2024-04-23T21:23:49.395000+0000","last_active":"2024-04-23T21:24:02.436735+0000","last_peered":"2024-04-23T21:24:02.436735+0000","last_clean":"2024-04-23T21:24:02.436735+0000","last_became_active":"2024-04-23T21:23:49.313320+0000","last_became_peered":"2024-04-23T21:23:49.313320+0000","last_unstale":"2024-04-23T21:24:02.436735+0000","last_undegraded":"2024-04-23T21:24:02.436735+0000","last_fullsized":"2024-04-23T21:24:02.436735+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:00.953641+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.064094391000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3731,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436764+0000","last_change":"2024-04-23T21:23:49.438136+0000","last_active":"2024-04-23T21:24:02.436764+0000","last_peered":"2024-04-23T21:24:02.436764+0000","last_clean":"2024-04-23T21:24:02.436764+0000","last_became_active":"2024-04-23T21:23:49.313703+0000","last_became_peered":"2024-04-23T21:23:49.313703+0000","last_unstale":"2024-04-23T21:24:02.436764+0000","last_undegraded":"2024-04-23T21:24:02.436764+0000","last_fullsized":"2024-04-23T21:24:02.436764+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:02.636441+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.063117355999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417056+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417056+0000","last_peered":"2024-04-23T21:24:07.417056+0000","last_clean":"2024-04-23T21:24:07.417056+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417056+0000","last_undegraded":"2024-04-23T21:24:07.417056+0000","last_fullsized":"2024-04-23T21:24:07.417056+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:25:04.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3494,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416076+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416076+0000","last_peered":"2024-04-23T21:24:07.416076+0000","last_clean":"2024-04-23T21:24:07.416076+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416076+0000","last_undegraded":"2024-04-23T21:24:07.416076+0000","last_fullsized":"2024-04-23T21:24:07.416076+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:36.032019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5325,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-04-23T21:24:06.409211+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:06.409211+0000","last_peered":"2024-04-23T21:24:06.409211+0000","last_clean":"2024-04-23T21:24:06.409211+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:06.409211+0000","last_undegraded":"2024-04-23T21:24:06.409211+0000","last_fullsized":"2024-04-23T21:24:06.409211+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3109,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436400+0000","last_change":"2024-04-23T21:23:53.295392+0000","last_active":"2024-04-23T21:24:02.436400+0000","last_peered":"2024-04-23T21:24:02.436400+0000","last_clean":"2024-04-23T21:24:02.436400+0000","last_became_active":"2024-04-23T21:23:53.206512+0000","last_became_peered":"2024-04-23T21:23:53.206512+0000","last_unstale":"2024-04-23T21:24:02.436400+0000","last_undegraded":"2024-04-23T21:24:02.436400+0000","last_fullsized":"2024-04-23T21:24:02.436400+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:20:23.215602+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.084884373999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417233+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417233+0000","last_peered":"2024-04-23T21:24:07.417233+0000","last_clean":"2024-04-23T21:24:07.417233+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417233+0000","last_undegraded":"2024-04-23T21:24:07.417233+0000","last_fullsized":"2024-04-23T21:24:07.417233+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:18:30.932913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8777,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416829+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416829+0000","last_peered":"2024-04-23T21:24:07.416829+0000","last_clean":"2024-04-23T21:24:07.416829+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416829+0000","last_undegraded":"2024-04-23T21:24:07.416829+0000","last_fullsized":"2024-04-23T21:24:07.416829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:59.559020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436249+0000","last_change":"2024-04-23T21:23:51.305630+0000","last_active":"2024-04-23T21:24:02.436249+0000","last_peered":"2024-04-23T21:24:02.436249+0000","last_clean":"2024-04-23T21:24:02.436249+0000","last_became_active":"2024-04-23T21:23:51.305331+0000","last_became_peered":"2024-04-23T21:23:51.305331+0000","last_unstale":"2024-04-23T21:24:02.436249+0000","last_undegraded":"2024-04-23T21:24:02.436249+0000","last_fullsized":"2024-04-23T21:24:02.436249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:27:24.622720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2188,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436258+0000","last_change":"2024-04-23T21:23:53.287328+0000","last_active":"2024-04-23T21:24:02.436258+0000","last_peered":"2024-04-23T21:24:02.436258+0000","last_clean":"2024-04-23T21:24:02.436258+0000","last_became_active":"2024-04-23T21:23:53.206462+0000","last_became_peered":"2024-04-23T21:23:53.206462+0000","last_unstale":"2024-04-23T21:24:02.436258+0000","last_undegraded":"2024-04-23T21:24:02.436258+0000","last_fullsized":"2024-04-23T21:24:02.436258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:58.138448+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.079455974999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4155,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416423+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416423+0000","last_peered":"2024-04-23T21:24:07.416423+0000","last_clean":"2024-04-23T21:24:07.416423+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416423+0000","last_undegraded":"2024-04-23T21:24:07.416423+0000","last_fullsized":"2024-04-23T21:24:07.416423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:49.518562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865478,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30940,"kb_used_data":2060,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340900,"statfs":{"total":96636764160,"available":96605081600,"internally_reserved":0,"allocated":2109440,"data_stored":1034886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":2112,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":2162688,"data_stored":1625273,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:10.805 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:11.087 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:11.087 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:11.099 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-23T21:24:09.627366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380372,"kb_used_data":16672,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594348,"statfs":{"total":773094113280,"available":772704612352,"internally_reserved":0,"allocated":17072128,"data_stored":9494914,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244093"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8752,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436735+0000","last_change":"2024-04-23T21:23:49.395000+0000","last_active":"2024-04-23T21:24:02.436735+0000","last_peered":"2024-04-23T21:24:02.436735+0000","last_clean":"2024-04-23T21:24:02.436735+0000","last_became_active":"2024-04-23T21:23:49.313320+0000","last_became_peered":"2024-04-23T21:23:49.313320+0000","last_unstale":"2024-04-23T21:24:02.436735+0000","last_undegraded":"2024-04-23T21:24:02.436735+0000","last_fullsized":"2024-04-23T21:24:02.436735+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:00.953641+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.064094391000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3731,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436764+0000","last_change":"2024-04-23T21:23:49.438136+0000","last_active":"2024-04-23T21:24:02.436764+0000","last_peered":"2024-04-23T21:24:02.436764+0000","last_clean":"2024-04-23T21:24:02.436764+0000","last_became_active":"2024-04-23T21:23:49.313703+0000","last_became_peered":"2024-04-23T21:23:49.313703+0000","last_unstale":"2024-04-23T21:24:02.436764+0000","last_undegraded":"2024-04-23T21:24:02.436764+0000","last_fullsized":"2024-04-23T21:24:02.436764+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:02.636441+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.063117355999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417056+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417056+0000","last_peered":"2024-04-23T21:24:07.417056+0000","last_clean":"2024-04-23T21:24:07.417056+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417056+0000","last_undegraded":"2024-04-23T21:24:07.417056+0000","last_fullsized":"2024-04-23T21:24:07.417056+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:25:04.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3494,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416076+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416076+0000","last_peered":"2024-04-23T21:24:07.416076+0000","last_clean":"2024-04-23T21:24:07.416076+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416076+0000","last_undegraded":"2024-04-23T21:24:07.416076+0000","last_fullsized":"2024-04-23T21:24:07.416076+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:36.032019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5325,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-04-23T21:24:06.409211+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:06.409211+0000","last_peered":"2024-04-23T21:24:06.409211+0000","last_clean":"2024-04-23T21:24:06.409211+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:06.409211+0000","last_undegraded":"2024-04-23T21:24:06.409211+0000","last_fullsized":"2024-04-23T21:24:06.409211+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3109,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436400+0000","last_change":"2024-04-23T21:23:53.295392+0000","last_active":"2024-04-23T21:24:02.436400+0000","last_peered":"2024-04-23T21:24:02.436400+0000","last_clean":"2024-04-23T21:24:02.436400+0000","last_became_active":"2024-04-23T21:23:53.206512+0000","last_became_peered":"2024-04-23T21:23:53.206512+0000","last_unstale":"2024-04-23T21:24:02.436400+0000","last_undegraded":"2024-04-23T21:24:02.436400+0000","last_fullsized":"2024-04-23T21:24:02.436400+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:20:23.215602+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.084884373999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417233+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417233+0000","last_peered":"2024-04-23T21:24:07.417233+0000","last_clean":"2024-04-23T21:24:07.417233+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417233+0000","last_undegraded":"2024-04-23T21:24:07.417233+0000","last_fullsized":"2024-04-23T21:24:07.417233+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:18:30.932913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8777,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416829+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416829+0000","last_peered":"2024-04-23T21:24:07.416829+0000","last_clean":"2024-04-23T21:24:07.416829+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416829+0000","last_undegraded":"2024-04-23T21:24:07.416829+0000","last_fullsized":"2024-04-23T21:24:07.416829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:59.559020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436249+0000","last_change":"2024-04-23T21:23:51.305630+0000","last_active":"2024-04-23T21:24:02.436249+0000","last_peered":"2024-04-23T21:24:02.436249+0000","last_clean":"2024-04-23T21:24:02.436249+0000","last_became_active":"2024-04-23T21:23:51.305331+0000","last_became_peered":"2024-04-23T21:23:51.305331+0000","last_unstale":"2024-04-23T21:24:02.436249+0000","last_undegraded":"2024-04-23T21:24:02.436249+0000","last_fullsized":"2024-04-23T21:24:02.436249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:27:24.622720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2188,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436258+0000","last_change":"2024-04-23T21:23:53.287328+0000","last_active":"2024-04-23T21:24:02.436258+0000","last_peered":"2024-04-23T21:24:02.436258+0000","last_clean":"2024-04-23T21:24:02.436258+0000","last_became_active":"2024-04-23T21:23:53.206462+0000","last_became_peered":"2024-04-23T21:23:53.206462+0000","last_unstale":"2024-04-23T21:24:02.436258+0000","last_undegraded":"2024-04-23T21:24:02.436258+0000","last_fullsized":"2024-04-23T21:24:02.436258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:58.138448+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.079455974999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4155,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416423+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416423+0000","last_peered":"2024-04-23T21:24:07.416423+0000","last_clean":"2024-04-23T21:24:07.416423+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416423+0000","last_undegraded":"2024-04-23T21:24:07.416423+0000","last_fullsized":"2024-04-23T21:24:07.416423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:49.518562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865478,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30940,"kb_used_data":2060,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340900,"statfs":{"total":96636764160,"available":96605081600,"internally_reserved":0,"allocated":2109440,"data_stored":1034886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":2112,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":2162688,"data_stored":1625273,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:11.101 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-23T21:24:11.101 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:11.384 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:11.385 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:11.397 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-04-23T21:24:09.627366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380372,"kb_used_data":16672,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594348,"statfs":{"total":773094113280,"available":772704612352,"internally_reserved":0,"allocated":17072128,"data_stored":9494914,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.244093"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8752,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436735+0000","last_change":"2024-04-23T21:23:49.395000+0000","last_active":"2024-04-23T21:24:02.436735+0000","last_peered":"2024-04-23T21:24:02.436735+0000","last_clean":"2024-04-23T21:24:02.436735+0000","last_became_active":"2024-04-23T21:23:49.313320+0000","last_became_peered":"2024-04-23T21:23:49.313320+0000","last_unstale":"2024-04-23T21:24:02.436735+0000","last_undegraded":"2024-04-23T21:24:02.436735+0000","last_fullsized":"2024-04-23T21:24:02.436735+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:24:00.953641+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.064094391000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3731,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436764+0000","last_change":"2024-04-23T21:23:49.438136+0000","last_active":"2024-04-23T21:24:02.436764+0000","last_peered":"2024-04-23T21:24:02.436764+0000","last_clean":"2024-04-23T21:24:02.436764+0000","last_became_active":"2024-04-23T21:23:49.313703+0000","last_became_peered":"2024-04-23T21:23:49.313703+0000","last_unstale":"2024-04-23T21:24:02.436764+0000","last_undegraded":"2024-04-23T21:24:02.436764+0000","last_fullsized":"2024-04-23T21:24:02.436764+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:20:02.636441+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.063117355999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417056+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417056+0000","last_peered":"2024-04-23T21:24:07.417056+0000","last_clean":"2024-04-23T21:24:07.417056+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417056+0000","last_undegraded":"2024-04-23T21:24:07.417056+0000","last_fullsized":"2024-04-23T21:24:07.417056+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:25:04.673221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3494,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416076+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416076+0000","last_peered":"2024-04-23T21:24:07.416076+0000","last_clean":"2024-04-23T21:24:07.416076+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416076+0000","last_undegraded":"2024-04-23T21:24:07.416076+0000","last_fullsized":"2024-04-23T21:24:07.416076+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:11:36.032019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5325,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-04-23T21:24:06.409211+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:06.409211+0000","last_peered":"2024-04-23T21:24:06.409211+0000","last_clean":"2024-04-23T21:24:06.409211+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:06.409211+0000","last_undegraded":"2024-04-23T21:24:06.409211+0000","last_fullsized":"2024-04-23T21:24:06.409211+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3109,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436400+0000","last_change":"2024-04-23T21:23:53.295392+0000","last_active":"2024-04-23T21:24:02.436400+0000","last_peered":"2024-04-23T21:24:02.436400+0000","last_clean":"2024-04-23T21:24:02.436400+0000","last_became_active":"2024-04-23T21:23:53.206512+0000","last_became_peered":"2024-04-23T21:23:53.206512+0000","last_unstale":"2024-04-23T21:24:02.436400+0000","last_undegraded":"2024-04-23T21:24:02.436400+0000","last_fullsized":"2024-04-23T21:24:02.436400+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:20:23.215602+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.084884373999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4794,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417233+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417233+0000","last_peered":"2024-04-23T21:24:07.417233+0000","last_clean":"2024-04-23T21:24:07.417233+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417233+0000","last_undegraded":"2024-04-23T21:24:07.417233+0000","last_fullsized":"2024-04-23T21:24:07.417233+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:18:30.932913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8777,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416829+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416829+0000","last_peered":"2024-04-23T21:24:07.416829+0000","last_clean":"2024-04-23T21:24:07.416829+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416829+0000","last_undegraded":"2024-04-23T21:24:07.416829+0000","last_fullsized":"2024-04-23T21:24:07.416829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:59.559020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436249+0000","last_change":"2024-04-23T21:23:51.305630+0000","last_active":"2024-04-23T21:24:02.436249+0000","last_peered":"2024-04-23T21:24:02.436249+0000","last_clean":"2024-04-23T21:24:02.436249+0000","last_became_active":"2024-04-23T21:23:51.305331+0000","last_became_peered":"2024-04-23T21:23:51.305331+0000","last_unstale":"2024-04-23T21:24:02.436249+0000","last_undegraded":"2024-04-23T21:24:02.436249+0000","last_fullsized":"2024-04-23T21:24:02.436249+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:27:24.622720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2188,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-04-23T21:24:02.436258+0000","last_change":"2024-04-23T21:23:53.287328+0000","last_active":"2024-04-23T21:24:02.436258+0000","last_peered":"2024-04-23T21:24:02.436258+0000","last_clean":"2024-04-23T21:24:02.436258+0000","last_became_active":"2024-04-23T21:23:53.206462+0000","last_became_peered":"2024-04-23T21:23:53.206462+0000","last_unstale":"2024-04-23T21:24:02.436258+0000","last_undegraded":"2024-04-23T21:24:02.436258+0000","last_fullsized":"2024-04-23T21:24:02.436258+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:54:58.138448+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.079455974999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4155,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416423+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416423+0000","last_peered":"2024-04-23T21:24:07.416423+0000","last_clean":"2024-04-23T21:24:07.416423+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416423+0000","last_undegraded":"2024-04-23T21:24:07.416423+0000","last_fullsized":"2024-04-23T21:24:07.416423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:31:49.518562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865478,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30940,"kb_used_data":2060,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340900,"statfs":{"total":96636764160,"available":96605081600,"internally_reserved":0,"allocated":2109440,"data_stored":1034886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528350,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30476,"kb_used_data":2112,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341364,"statfs":{"total":96636764160,"available":96605556736,"internally_reserved":0,"allocated":2162688,"data_stored":1625273,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:11.399 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-23T21:24:11.682 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:11.682 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":176,"fsid":"3556a001-ee44-4b6e-8d50-1bc45ce74758","created":"2024-04-23T21:20:57.391123+0000","modified":"2024-04-23T21:24:07.406390+0000","last_up_change":"2024-04-23T21:23:48.290514+0000","last_in_change":"2024-04-23T21:23:50.294338+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-23T21:21:08.380744+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-23T21:21:09.608446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-04-23T21:21:16.010288+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":51,"snap_epoch":152,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"30c47d3d-583d-4987-9b86-247b80cb80ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":163,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6809","nonce":1367522096}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6811","nonce":1367522096}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6815","nonce":1367522096}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1367522096},{"type":"v1","addr":"172.21.15.111:6813","nonce":1367522096}]},"public_addr":"172.21.15.111:6809/1367522096","cluster_addr":"172.21.15.111:6811/1367522096","heartbeat_back_addr":"172.21.15.111:6815/1367522096","heartbeat_front_addr":"172.21.15.111:6813/1367522096","state":["exists","up"]},{"osd":1,"uuid":"d7c695c5-c361-4c22-b4e7-bd16572f0550","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":17,"up_from":45,"up_thru":173,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6817","nonce":723341542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6819","nonce":723341542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6823","nonce":723341542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":723341542},{"type":"v1","addr":"172.21.15.111:6821","nonce":723341542}]},"public_addr":"172.21.15.111:6817/723341542","cluster_addr":"172.21.15.111:6819/723341542","heartbeat_back_addr":"172.21.15.111:6823/723341542","heartbeat_front_addr":"172.21.15.111:6821/723341542","state":["exists","up"]},{"osd":2,"uuid":"46e4a0e9-b403-4a09-92ca-57245c3fec30","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6801","nonce":698077721}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6803","nonce":698077721}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6807","nonce":698077721}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":698077721},{"type":"v1","addr":"172.21.15.111:6805","nonce":698077721}]},"public_addr":"172.21.15.111:6801/698077721","cluster_addr":"172.21.15.111:6803/698077721","heartbeat_back_addr":"172.21.15.111:6807/698077721","heartbeat_front_addr":"172.21.15.111:6805/698077721","state":["exists","up"]},{"osd":3,"uuid":"c18de0c4-3fd5-46a0-85a6-f813c8202ea8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6825","nonce":4030519478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6827","nonce":4030519478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6831","nonce":4030519478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":4030519478},{"type":"v1","addr":"172.21.15.111:6829","nonce":4030519478}]},"public_addr":"172.21.15.111:6825/4030519478","cluster_addr":"172.21.15.111:6827/4030519478","heartbeat_back_addr":"172.21.15.111:6831/4030519478","heartbeat_front_addr":"172.21.15.111:6829/4030519478","state":["exists","up"]},{"osd":4,"uuid":"d0562ad9-c051-475a-a40b-248d222c8921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6817","nonce":1513005997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6819","nonce":1513005997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6824","nonce":1513005997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1513005997},{"type":"v1","addr":"172.21.15.192:6821","nonce":1513005997}]},"public_addr":"172.21.15.192:6817/1513005997","cluster_addr":"172.21.15.192:6819/1513005997","heartbeat_back_addr":"172.21.15.192:6824/1513005997","heartbeat_front_addr":"172.21.15.192:6821/1513005997","state":["exists","up"]},{"osd":5,"uuid":"344f8beb-b979-4248-92df-c34cd560be6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":104,"up_from":157,"up_thru":175,"down_at":105,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6801","nonce":1231536685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6803","nonce":1231536685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6807","nonce":1231536685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1231536685},{"type":"v1","addr":"172.21.15.192:6805","nonce":1231536685}]},"public_addr":"172.21.15.192:6801/1231536685","cluster_addr":"172.21.15.192:6803/1231536685","heartbeat_back_addr":"172.21.15.192:6807/1231536685","heartbeat_front_addr":"172.21.15.192:6805/1231536685","state":["exists","up"]},{"osd":6,"uuid":"62434311-7dc6-4a3e-bd07-1e8915b37709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":160,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6823","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6825","nonce":2824610433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6827","nonce":2824610433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6831","nonce":2824610433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":2824610433},{"type":"v1","addr":"172.21.15.192:6829","nonce":2824610433}]},"public_addr":"172.21.15.192:6825/2824610433","cluster_addr":"172.21.15.192:6827/2824610433","heartbeat_back_addr":"172.21.15.192:6831/2824610433","heartbeat_front_addr":"172.21.15.192:6829/2824610433","state":["exists","up"]},{"osd":7,"uuid":"db260d92-d3d7-464e-ab34-0bedd2991992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6809","nonce":1007335403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6811","nonce":1007335403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6815","nonce":1007335403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1007335403},{"type":"v1","addr":"172.21.15.192:6813","nonce":1007335403}]},"public_addr":"172.21.15.192:6809/1007335403","cluster_addr":"172.21.15.192:6811/1007335403","heartbeat_back_addr":"172.21.15.192:6815/1007335403","heartbeat_front_addr":"172.21.15.192:6813/1007335403","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.914033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.903496+0000","dead_epoch":18},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.934930+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:05.001485+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.979253+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.839146+0000","dead_epoch":105},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.963622+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T21:21:04.946000+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.1f","osds":[2,3]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T21:24:12.696 INFO:tasks.ceph:Scrubbing osd.0 2024-04-23T21:24:12.696 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:12.830 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:12.830 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:12.830 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:12.841 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-23T21:24:13.123 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 0 to deep-scrub 2024-04-23T21:24:13.136 INFO:tasks.ceph:Scrubbing osd.1 2024-04-23T21:24:13.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:13.257 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:13.258 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:13.258 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:13.268 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-23T21:24:13.555 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 1 to deep-scrub 2024-04-23T21:24:13.568 INFO:tasks.ceph:Scrubbing osd.2 2024-04-23T21:24:13.568 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:13.690 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:13.690 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:13.690 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:13.701 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-23T21:24:13.989 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 2 to deep-scrub 2024-04-23T21:24:14.002 INFO:tasks.ceph:Scrubbing osd.3 2024-04-23T21:24:14.002 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:14.125 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:14.125 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:14.125 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:14.136 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-23T21:24:14.421 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 3 to deep-scrub 2024-04-23T21:24:14.434 INFO:tasks.ceph:Scrubbing osd.4 2024-04-23T21:24:14.435 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:14.559 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:14.559 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:14.559 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:14.570 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-23T21:24:14.855 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 4 to deep-scrub 2024-04-23T21:24:14.868 INFO:tasks.ceph:Scrubbing osd.5 2024-04-23T21:24:14.868 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:14.992 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:14.993 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-04-23T21:24:14.993 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:15.006 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-23T21:24:15.288 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 5 to deep-scrub 2024-04-23T21:24:15.301 INFO:tasks.ceph:Scrubbing osd.6 2024-04-23T21:24:15.302 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:15.425 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:15.425 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:15.425 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:15.436 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-23T21:24:15.722 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 6 to deep-scrub 2024-04-23T21:24:15.733 INFO:tasks.ceph:Scrubbing osd.7 2024-04-23T21:24:15.734 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-23T21:24:15.855 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-23T21:24:15.855 INFO:teuthology.orchestra.run.smithi111.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-23T21:24:15.855 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-23T21:24:15.866 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-23T21:24:16.161 INFO:teuthology.orchestra.run.smithi111.stderr:instructed osd(s) 7 to deep-scrub 2024-04-23T21:24:16.174 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:16.461 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:16.462 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:16.473 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-04-23T21:24:15.628568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18338,"ondisk_log_size":18338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":380404,"kb_used_data":16696,"kb_used_omap":47,"kb_used_meta":363536,"kb_avail":754594316,"statfs":{"total":773094113280,"available":772704579584,"internally_reserved":0,"allocated":17096704,"data_stored":9506598,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":372261643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.221553"},"pg_stats":[{"pgid":"3.1f","version":"152'521","reported_seq":5057,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418462+0000","last_change":"2024-04-23T21:23:53.209462+0000","last_active":"2024-04-23T21:24:07.418462+0000","last_peered":"2024-04-23T21:24:07.418462+0000","last_clean":"2024-04-23T21:24:07.418462+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:07.418462+0000","last_undegraded":"2024-04-23T21:24:07.418462+0000","last_fullsized":"2024-04-23T21:24:07.418462+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":521,"log_dups_size":0,"ondisk_log_size":521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:10.911974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"151'466","reported_seq":7105,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414567+0000","last_change":"2024-04-23T21:23:53.207941+0000","last_active":"2024-04-23T21:24:07.414567+0000","last_peered":"2024-04-23T21:24:07.414567+0000","last_clean":"2024-04-23T21:24:07.414567+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:07.414567+0000","last_undegraded":"2024-04-23T21:24:07.414567+0000","last_fullsized":"2024-04-23T21:24:07.414567+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:26:57.118854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"158'347","reported_seq":8733,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416120+0000","last_change":"2024-04-23T21:24:07.416120+0000","last_active":"2024-04-23T21:24:07.416120+0000","last_peered":"2024-04-23T21:24:07.416120+0000","last_clean":"2024-04-23T21:24:07.416120+0000","last_became_active":"2024-04-23T21:24:07.413798+0000","last_became_peered":"2024-04-23T21:24:07.413798+0000","last_unstale":"2024-04-23T21:24:07.416120+0000","last_undegraded":"2024-04-23T21:24:07.416120+0000","last_fullsized":"2024-04-23T21:24:07.416120+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:12.168860+0000","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.061760584,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"158'527","reported_seq":3688,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416115+0000","last_change":"2024-04-23T21:24:07.416115+0000","last_active":"2024-04-23T21:24:07.416115+0000","last_peered":"2024-04-23T21:24:07.416115+0000","last_clean":"2024-04-23T21:24:07.416115+0000","last_became_active":"2024-04-23T21:24:07.413677+0000","last_became_peered":"2024-04-23T21:24:07.413677+0000","last_unstale":"2024-04-23T21:24:07.416115+0000","last_undegraded":"2024-04-23T21:24:07.416115+0000","last_fullsized":"2024-04-23T21:24:07.416115+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:33:35.515414+0000","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.056714265,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"152'434","reported_seq":11133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417096+0000","last_change":"2024-04-23T21:23:53.211307+0000","last_active":"2024-04-23T21:24:07.417096+0000","last_peered":"2024-04-23T21:24:07.417096+0000","last_clean":"2024-04-23T21:24:07.417096+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:07.417096+0000","last_undegraded":"2024-04-23T21:24:07.417096+0000","last_fullsized":"2024-04-23T21:24:07.417096+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":434,"log_dups_size":0,"ondisk_log_size":434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"151'931","reported_seq":6817,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412777+0000","last_change":"2024-04-23T21:23:36.267346+0000","last_active":"2024-04-23T21:24:07.412777+0000","last_peered":"2024-04-23T21:24:07.412777+0000","last_clean":"2024-04-23T21:24:07.412777+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:07.412777+0000","last_undegraded":"2024-04-23T21:24:07.412777+0000","last_fullsized":"2024-04-23T21:24:07.412777+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:25:31.378293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"151'880","reported_seq":3552,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420989+0000","last_change":"2024-04-23T21:23:54.206463+0000","last_active":"2024-04-23T21:24:07.420989+0000","last_peered":"2024-04-23T21:24:07.420989+0000","last_clean":"2024-04-23T21:24:07.420989+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:07.420989+0000","last_undegraded":"2024-04-23T21:24:07.420989+0000","last_fullsized":"2024-04-23T21:24:07.420989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:27:05.909209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"152'742","reported_seq":6255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414771+0000","last_change":"2024-04-23T21:23:54.208832+0000","last_active":"2024-04-23T21:24:07.414771+0000","last_peered":"2024-04-23T21:24:07.414771+0000","last_clean":"2024-04-23T21:24:07.414771+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:07.414771+0000","last_undegraded":"2024-04-23T21:24:07.414771+0000","last_fullsized":"2024-04-23T21:24:07.414771+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:47:05.777566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"152'612","reported_seq":3495,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416130+0000","last_change":"2024-04-23T21:23:53.269592+0000","last_active":"2024-04-23T21:24:07.416130+0000","last_peered":"2024-04-23T21:24:07.416130+0000","last_clean":"2024-04-23T21:24:07.416130+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:07.416130+0000","last_undegraded":"2024-04-23T21:24:07.416130+0000","last_fullsized":"2024-04-23T21:24:07.416130+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"143'461","reported_seq":5327,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.410590+0000","last_change":"2024-04-23T21:23:50.354393+0000","last_active":"2024-04-23T21:24:07.410590+0000","last_peered":"2024-04-23T21:24:07.410590+0000","last_clean":"2024-04-23T21:24:07.410590+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:07.410590+0000","last_undegraded":"2024-04-23T21:24:07.410590+0000","last_fullsized":"2024-04-23T21:24:07.410590+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:17:07.344883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"152'411","reported_seq":5178,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416658+0000","last_change":"2024-04-23T21:23:52.211847+0000","last_active":"2024-04-23T21:24:07.416658+0000","last_peered":"2024-04-23T21:24:07.416658+0000","last_clean":"2024-04-23T21:24:07.416658+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:07.416658+0000","last_undegraded":"2024-04-23T21:24:07.416658+0000","last_fullsized":"2024-04-23T21:24:07.416658+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:56:17.262731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"152'693","reported_seq":6678,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412884+0000","last_change":"2024-04-23T21:23:53.203475+0000","last_active":"2024-04-23T21:24:07.412884+0000","last_peered":"2024-04-23T21:24:07.412884+0000","last_clean":"2024-04-23T21:24:07.412884+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:07.412884+0000","last_undegraded":"2024-04-23T21:24:07.412884+0000","last_fullsized":"2024-04-23T21:24:07.412884+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":693,"log_dups_size":0,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:06:39.051714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":344,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411712+0000","last_change":"2024-04-23T21:23:50.315552+0000","last_active":"2024-04-23T21:24:07.411712+0000","last_peered":"2024-04-23T21:24:07.411712+0000","last_clean":"2024-04-23T21:24:07.411712+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:07.411712+0000","last_undegraded":"2024-04-23T21:24:07.411712+0000","last_fullsized":"2024-04-23T21:24:07.411712+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:03:07.837444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"162'489","reported_seq":3099,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.424872+0000","last_change":"2024-04-23T21:24:07.424872+0000","last_active":"2024-04-23T21:24:07.424872+0000","last_peered":"2024-04-23T21:24:07.424872+0000","last_clean":"2024-04-23T21:24:07.424872+0000","last_became_active":"2024-04-23T21:24:07.422375+0000","last_became_peered":"2024-04-23T21:24:07.422375+0000","last_unstale":"2024-04-23T21:24:07.424872+0000","last_undegraded":"2024-04-23T21:24:07.424872+0000","last_fullsized":"2024-04-23T21:24:07.424872+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:39:24.396706+0000","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.079589640000000003,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":94,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411520+0000","last_change":"2024-04-23T21:23:51.306559+0000","last_active":"2024-04-23T21:24:07.411520+0000","last_peered":"2024-04-23T21:24:07.411520+0000","last_clean":"2024-04-23T21:24:07.411520+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:07.411520+0000","last_undegraded":"2024-04-23T21:24:07.411520+0000","last_fullsized":"2024-04-23T21:24:07.411520+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:15:03.798947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"152'468","reported_seq":4639,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.420867+0000","last_change":"2024-04-23T21:23:36.258340+0000","last_active":"2024-04-23T21:24:07.420867+0000","last_peered":"2024-04-23T21:24:07.420867+0000","last_clean":"2024-04-23T21:24:07.420867+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:07.420867+0000","last_undegraded":"2024-04-23T21:24:07.420867+0000","last_fullsized":"2024-04-23T21:24:07.420867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:27:22.920498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":332,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412980+0000","last_change":"2024-04-23T21:21:09.797976+0000","last_active":"2024-04-23T21:24:07.412980+0000","last_peered":"2024-04-23T21:24:07.412980+0000","last_clean":"2024-04-23T21:24:07.412980+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:07.412980+0000","last_undegraded":"2024-04-23T21:24:07.412980+0000","last_fullsized":"2024-04-23T21:24:07.412980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:36.108086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"151'438","reported_seq":3961,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414683+0000","last_change":"2024-04-23T21:23:52.215671+0000","last_active":"2024-04-23T21:24:07.414683+0000","last_peered":"2024-04-23T21:24:07.414683+0000","last_clean":"2024-04-23T21:24:07.414683+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:07.414683+0000","last_undegraded":"2024-04-23T21:24:07.414683+0000","last_fullsized":"2024-04-23T21:24:07.414683+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:44.434861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":40,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414667+0000","last_change":"2024-04-23T21:23:50.309190+0000","last_active":"2024-04-23T21:24:07.414667+0000","last_peered":"2024-04-23T21:24:07.414667+0000","last_clean":"2024-04-23T21:24:07.414667+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:07.414667+0000","last_undegraded":"2024-04-23T21:24:07.414667+0000","last_fullsized":"2024-04-23T21:24:07.414667+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:03:50.612599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"152'427","reported_seq":4795,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.417275+0000","last_change":"2024-04-23T21:23:55.206788+0000","last_active":"2024-04-23T21:24:07.417275+0000","last_peered":"2024-04-23T21:24:07.417275+0000","last_clean":"2024-04-23T21:24:07.417275+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:07.417275+0000","last_undegraded":"2024-04-23T21:24:07.417275+0000","last_fullsized":"2024-04-23T21:24:07.417275+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":427,"log_dups_size":0,"ondisk_log_size":427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"152'423","reported_seq":8778,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416877+0000","last_change":"2024-04-23T21:23:53.208780+0000","last_active":"2024-04-23T21:24:07.416877+0000","last_peered":"2024-04-23T21:24:07.416877+0000","last_clean":"2024-04-23T21:24:07.416877+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:07.416877+0000","last_undegraded":"2024-04-23T21:24:07.416877+0000","last_fullsized":"2024-04-23T21:24:07.416877+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":442,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416105+0000","last_change":"2024-04-23T21:23:50.314034+0000","last_active":"2024-04-23T21:24:07.416105+0000","last_peered":"2024-04-23T21:24:07.416105+0000","last_clean":"2024-04-23T21:24:07.416105+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:07.416105+0000","last_undegraded":"2024-04-23T21:24:07.416105+0000","last_fullsized":"2024-04-23T21:24:07.416105+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:09.720609+0000","last_clean_scrub_stamp":"2024-04-23T21:21:09.720609+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:08:06.071432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":246,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421322+0000","last_change":"2024-04-23T21:24:05.473187+0000","last_active":"2024-04-23T21:24:07.421322+0000","last_peered":"2024-04-23T21:24:07.421322+0000","last_clean":"2024-04-23T21:24:07.421322+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:07.421322+0000","last_undegraded":"2024-04-23T21:24:07.421322+0000","last_fullsized":"2024-04-23T21:24:07.421322+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:04:14.957195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"152'337","reported_seq":1390,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.421156+0000","last_change":"2024-04-23T21:23:36.259228+0000","last_active":"2024-04-23T21:24:07.421156+0000","last_peered":"2024-04-23T21:24:07.421156+0000","last_clean":"2024-04-23T21:24:07.421156+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:07.421156+0000","last_undegraded":"2024-04-23T21:24:07.421156+0000","last_fullsized":"2024-04-23T21:24:07.421156+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:12:14.868722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416426+0000","last_change":"2024-04-23T21:23:51.329262+0000","last_active":"2024-04-23T21:24:07.416426+0000","last_peered":"2024-04-23T21:24:07.416426+0000","last_clean":"2024-04-23T21:24:07.416426+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:07.416426+0000","last_undegraded":"2024-04-23T21:24:07.416426+0000","last_fullsized":"2024-04-23T21:24:07.416426+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:26:57.759771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"152'968","reported_seq":9513,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.413014+0000","last_change":"2024-04-23T21:23:54.213838+0000","last_active":"2024-04-23T21:24:07.413014+0000","last_peered":"2024-04-23T21:24:07.413014+0000","last_clean":"2024-04-23T21:24:07.413014+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:07.413014+0000","last_undegraded":"2024-04-23T21:24:07.413014+0000","last_fullsized":"2024-04-23T21:24:07.413014+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:08:13.512288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":185,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.418787+0000","last_change":"2024-04-23T21:24:07.418787+0000","last_active":"2024-04-23T21:24:07.418787+0000","last_peered":"2024-04-23T21:24:07.418787+0000","last_clean":"2024-04-23T21:24:07.418787+0000","last_became_active":"2024-04-23T21:24:07.418557+0000","last_became_peered":"2024-04-23T21:24:07.418557+0000","last_unstale":"2024-04-23T21:24:07.418787+0000","last_undegraded":"2024-04-23T21:24:07.418787+0000","last_fullsized":"2024-04-23T21:24:07.418787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:07:36.828565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"162'636","reported_seq":5562,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414352+0000","last_change":"2024-04-23T21:23:53.277685+0000","last_active":"2024-04-23T21:24:07.414352+0000","last_peered":"2024-04-23T21:24:07.414352+0000","last_clean":"2024-04-23T21:24:07.414352+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:07.414352+0000","last_undegraded":"2024-04-23T21:24:07.414352+0000","last_fullsized":"2024-04-23T21:24:07.414352+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":636,"log_dups_size":0,"ondisk_log_size":636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:37:35.961255+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416158+0000","last_change":"2024-04-23T21:23:51.328119+0000","last_active":"2024-04-23T21:24:07.416158+0000","last_peered":"2024-04-23T21:24:07.416158+0000","last_clean":"2024-04-23T21:24:07.416158+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:07.416158+0000","last_undegraded":"2024-04-23T21:24:07.416158+0000","last_fullsized":"2024-04-23T21:24:07.416158+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:08.710721+0000","last_clean_scrub_stamp":"2024-04-23T21:21:08.710721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:54:45.382070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"152'317","reported_seq":3251,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414470+0000","last_change":"2024-04-23T21:23:52.211954+0000","last_active":"2024-04-23T21:24:07.414470+0000","last_peered":"2024-04-23T21:24:07.414470+0000","last_clean":"2024-04-23T21:24:07.414470+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:07.414470+0000","last_undegraded":"2024-04-23T21:24:07.414470+0000","last_fullsized":"2024-04-23T21:24:07.414470+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:19:38.692257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"151'368","reported_seq":7685,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.412727+0000","last_change":"2024-04-23T21:23:36.269693+0000","last_active":"2024-04-23T21:24:07.412727+0000","last_peered":"2024-04-23T21:24:07.412727+0000","last_clean":"2024-04-23T21:24:07.412727+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:07.412727+0000","last_undegraded":"2024-04-23T21:24:07.412727+0000","last_fullsized":"2024-04-23T21:24:07.412727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:34:37.051706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"151'621","reported_seq":4306,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411494+0000","last_change":"2024-04-23T21:23:36.260556+0000","last_active":"2024-04-23T21:24:07.411494+0000","last_peered":"2024-04-23T21:24:07.411494+0000","last_clean":"2024-04-23T21:24:07.411494+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:07.411494+0000","last_undegraded":"2024-04-23T21:24:07.411494+0000","last_fullsized":"2024-04-23T21:24:07.411494+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:46:39.902868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"149'496","reported_seq":3012,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.414429+0000","last_change":"2024-04-23T21:23:54.209300+0000","last_active":"2024-04-23T21:24:07.414429+0000","last_peered":"2024-04-23T21:24:07.414429+0000","last_clean":"2024-04-23T21:24:07.414429+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:07.414429+0000","last_undegraded":"2024-04-23T21:24:07.414429+0000","last_fullsized":"2024-04-23T21:24:07.414429+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":496,"log_dups_size":0,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:35:01.343829+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"162'437","reported_seq":2170,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.423708+0000","last_change":"2024-04-23T21:24:07.423708+0000","last_active":"2024-04-23T21:24:07.423708+0000","last_peered":"2024-04-23T21:24:07.423708+0000","last_clean":"2024-04-23T21:24:07.423708+0000","last_became_active":"2024-04-23T21:24:07.422312+0000","last_became_peered":"2024-04-23T21:24:07.422312+0000","last_unstale":"2024-04-23T21:24:07.423708+0000","last_undegraded":"2024-04-23T21:24:07.423708+0000","last_fullsized":"2024-04-23T21:24:07.423708+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:48:42.449000+0000","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":0.075475576000000003,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"150'320","reported_seq":3127,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416753+0000","last_change":"2024-04-23T21:23:50.401474+0000","last_active":"2024-04-23T21:24:07.416753+0000","last_peered":"2024-04-23T21:24:07.416753+0000","last_clean":"2024-04-23T21:24:07.416753+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:07.416753+0000","last_undegraded":"2024-04-23T21:24:07.416753+0000","last_fullsized":"2024-04-23T21:24:07.416753+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T06:21:13.760929+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"150'635","reported_seq":2827,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416474+0000","last_change":"2024-04-23T21:23:53.202599+0000","last_active":"2024-04-23T21:24:07.416474+0000","last_peered":"2024-04-23T21:24:07.416474+0000","last_clean":"2024-04-23T21:24:07.416474+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:07.416474+0000","last_undegraded":"2024-04-23T21:24:07.416474+0000","last_fullsized":"2024-04-23T21:24:07.416474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:02:35.518121+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"151'1078","reported_seq":11106,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416268+0000","last_change":"2024-04-23T21:23:53.203390+0000","last_active":"2024-04-23T21:24:07.416268+0000","last_peered":"2024-04-23T21:24:07.416268+0000","last_clean":"2024-04-23T21:24:07.416268+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:07.416268+0000","last_undegraded":"2024-04-23T21:24:07.416268+0000","last_fullsized":"2024-04-23T21:24:07.416268+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":1078,"log_dups_size":0,"ondisk_log_size":1078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:04:26.281528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"146'549","reported_seq":3866,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.411460+0000","last_change":"2024-04-23T21:23:52.204165+0000","last_active":"2024-04-23T21:24:07.411460+0000","last_peered":"2024-04-23T21:24:07.411460+0000","last_clean":"2024-04-23T21:24:07.411460+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:07.411460+0000","last_undegraded":"2024-04-23T21:24:07.411460+0000","last_fullsized":"2024-04-23T21:24:07.411460+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:24:50.439326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"151'309","reported_seq":2841,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416293+0000","last_change":"2024-04-23T21:23:52.204340+0000","last_active":"2024-04-23T21:24:07.416293+0000","last_peered":"2024-04-23T21:24:07.416293+0000","last_clean":"2024-04-23T21:24:07.416293+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:07.416293+0000","last_undegraded":"2024-04-23T21:24:07.416293+0000","last_fullsized":"2024-04-23T21:24:07.416293+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:17:17.804446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"150'955","reported_seq":4156,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416477+0000","last_change":"2024-04-23T21:23:53.208985+0000","last_active":"2024-04-23T21:24:07.416477+0000","last_peered":"2024-04-23T21:24:07.416477+0000","last_clean":"2024-04-23T21:24:07.416477+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:07.416477+0000","last_undegraded":"2024-04-23T21:24:07.416477+0000","last_fullsized":"2024-04-23T21:24:07.416477+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":955,"log_dups_size":0,"ondisk_log_size":955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"149'925","reported_seq":3523,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:07.416307+0000","last_change":"2024-04-23T21:23:52.209260+0000","last_active":"2024-04-23T21:24:07.416307+0000","last_peered":"2024-04-23T21:24:07.416307+0000","last_clean":"2024-04-23T21:24:07.416307+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:07.416307+0000","last_undegraded":"2024-04-23T21:24:07.416307+0000","last_fullsized":"2024-04-23T21:24:07.416307+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:21:16.757901+0000","last_clean_scrub_stamp":"2024-04-23T21:21:16.757901+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:01:08.572557+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18221,"ondisk_log_size":18221,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63848,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307992,"statfs":{"total":96636764160,"available":96571383808,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":63175114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29100,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342740,"statfs":{"total":96636764160,"available":96606965760,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865479,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30956,"kb_used_data":2072,"kb_used_omap":31,"kb_used_meta":28832,"kb_avail":94340884,"statfs":{"total":96636764160,"available":96605065216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":29524672},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69352,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94302488,"statfs":{"total":96636764160,"available":96565747712,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41576,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330264,"statfs":{"total":96636764160,"available":96594190336,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30492,"kb_used_data":2124,"kb_used_omap":6,"kb_used_meta":28345,"kb_avail":94341348,"statfs":{"total":96636764160,"available":96605540352,"internally_reserved":0,"allocated":2174976,"data_stored":1631115,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":29025700},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59624,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312216,"statfs":{"total":96636764160,"available":96575709184,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:16.478 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.478 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.479 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.480 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-23T21:21:09.720609+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=9, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.481 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-23T21:21:08.710721+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=8, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.482 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-23T21:21:16.757901+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=21, tm_sec=16, tm_wday=1, tm_yday=114, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=23, tm_hour=21, tm_min=24, tm_sec=11, tm_wday=1, tm_yday=114, tm_isdst=0) 2024-04-23T21:24:16.483 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-23T21:24:36.484 DEBUG:teuthology.orchestra.run.smithi111:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-23T21:24:36.783 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-23T21:24:36.784 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-23T21:24:36.795 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-04-23T21:24:35.633000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":616,"num_read":191248,"num_read_kb":1022870,"num_write":20360,"num_write_kb":600677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18960,"ondisk_log_size":18960,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376184,"kb_used_data":16696,"kb_used_omap":47,"kb_used_meta":359376,"kb_avail":754598536,"statfs":{"total":773094113280,"available":772708900864,"internally_reserved":0,"allocated":17096704,"data_stored":9506598,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":48373,"internal_metadata":368001803},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002350"},"pg_stats":[{"pgid":"3.1f","version":"176'540","reported_seq":5085,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:14.602717+0000","last_change":"2024-04-23T21:24:14.602800+0000","last_active":"2024-04-23T21:24:14.602717+0000","last_peered":"2024-04-23T21:24:14.602717+0000","last_clean":"2024-04-23T21:24:14.602717+0000","last_became_active":"2024-04-23T21:23:53.207833+0000","last_became_peered":"2024-04-23T21:23:53.207833+0000","last_unstale":"2024-04-23T21:24:14.602717+0000","last_undegraded":"2024-04-23T21:24:14.602717+0000","last_fullsized":"2024-04-23T21:24:14.602717+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'540","last_scrub_stamp":"2024-04-23T21:24:14.602630+0000","last_deep_scrub":"176'540","last_deep_scrub_stamp":"2024-04-23T21:24:14.602630+0000","last_clean_scrub_stamp":"2024-04-23T21:24:14.602630+0000","objects_scrubbed":19,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:52:01.597372+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.001628591,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6834,"num_read_kb":17494,"num_write":551,"num_write_kb":33877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":187273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"176'485","reported_seq":7133,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:19.140283+0000","last_change":"2024-04-23T21:24:19.140374+0000","last_active":"2024-04-23T21:24:19.140283+0000","last_peered":"2024-04-23T21:24:19.140283+0000","last_clean":"2024-04-23T21:24:19.140283+0000","last_became_active":"2024-04-23T21:23:53.206206+0000","last_became_peered":"2024-04-23T21:23:53.206206+0000","last_unstale":"2024-04-23T21:24:19.140283+0000","last_undegraded":"2024-04-23T21:24:19.140283+0000","last_fullsized":"2024-04-23T21:24:19.140283+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'485","last_scrub_stamp":"2024-04-23T21:24:19.140183+0000","last_deep_scrub":"176'485","last_deep_scrub_stamp":"2024-04-23T21:24:19.140183+0000","last_clean_scrub_stamp":"2024-04-23T21:24:19.140183+0000","objects_scrubbed":19,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:47:16.304517+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.001330747,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":6484,"num_read_kb":47284,"num_write":488,"num_write_kb":24910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":12637256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"176'370","reported_seq":8765,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.719164+0000","last_change":"2024-04-23T21:24:16.719213+0000","last_active":"2024-04-23T21:24:16.719164+0000","last_peered":"2024-04-23T21:24:16.719164+0000","last_clean":"2024-04-23T21:24:16.719164+0000","last_became_active":"2024-04-23T21:24:07.413798+0000","last_became_peered":"2024-04-23T21:24:07.413798+0000","last_unstale":"2024-04-23T21:24:16.719164+0000","last_undegraded":"2024-04-23T21:24:16.719164+0000","last_fullsized":"2024-04-23T21:24:16.719164+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'370","last_scrub_stamp":"2024-04-23T21:24:16.719080+0000","last_deep_scrub":"176'370","last_deep_scrub_stamp":"2024-04-23T21:24:16.719080+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.719080+0000","objects_scrubbed":23,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:44:17.831192+0000","scrub_duration":16,"objects_trimmed":14,"snaptrim_duration":0.061760584,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":8492,"num_read_kb":31999,"num_write":297,"num_write_kb":7758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"176'550","reported_seq":3720,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.717042+0000","last_change":"2024-04-23T21:24:17.717128+0000","last_active":"2024-04-23T21:24:17.717042+0000","last_peered":"2024-04-23T21:24:17.717042+0000","last_clean":"2024-04-23T21:24:17.717042+0000","last_became_active":"2024-04-23T21:24:07.413677+0000","last_became_peered":"2024-04-23T21:24:07.413677+0000","last_unstale":"2024-04-23T21:24:17.717042+0000","last_undegraded":"2024-04-23T21:24:17.717042+0000","last_fullsized":"2024-04-23T21:24:17.717042+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'550","last_scrub_stamp":"2024-04-23T21:24:17.716935+0000","last_deep_scrub":"176'550","last_deep_scrub_stamp":"2024-04-23T21:24:17.716935+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.716935+0000","objects_scrubbed":23,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:39:01.389550+0000","scrub_duration":15,"objects_trimmed":28,"snaptrim_duration":0.056714265,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":22,"num_read":3311,"num_read_kb":25049,"num_write":605,"num_write_kb":16878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"176'451","reported_seq":11158,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.899916+0000","last_change":"2024-04-23T21:24:15.899982+0000","last_active":"2024-04-23T21:24:15.899916+0000","last_peered":"2024-04-23T21:24:15.899916+0000","last_clean":"2024-04-23T21:24:15.899916+0000","last_became_active":"2024-04-23T21:23:53.203430+0000","last_became_peered":"2024-04-23T21:23:53.203430+0000","last_unstale":"2024-04-23T21:24:15.899916+0000","last_undegraded":"2024-04-23T21:24:15.899916+0000","last_fullsized":"2024-04-23T21:24:15.899916+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'451","last_scrub_stamp":"2024-04-23T21:24:15.899817+0000","last_deep_scrub":"176'451","last_deep_scrub_stamp":"2024-04-23T21:24:15.899817+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.899817+0000","objects_scrubbed":17,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:54:19.076326+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0039581360000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":11590,"num_read_kb":51962,"num_write":392,"num_write_kb":56494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":12778738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"176'952","reported_seq":6847,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.346400+0000","last_change":"2024-04-23T21:24:17.346474+0000","last_active":"2024-04-23T21:24:17.346400+0000","last_peered":"2024-04-23T21:24:17.346400+0000","last_clean":"2024-04-23T21:24:17.346400+0000","last_became_active":"2024-04-23T21:21:52.958009+0000","last_became_peered":"2024-04-23T21:21:52.958009+0000","last_unstale":"2024-04-23T21:24:17.346400+0000","last_undegraded":"2024-04-23T21:24:17.346400+0000","last_fullsized":"2024-04-23T21:24:17.346400+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'952","last_scrub_stamp":"2024-04-23T21:24:17.346309+0000","last_deep_scrub":"176'952","last_deep_scrub_stamp":"2024-04-23T21:24:17.346309+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.346309+0000","objects_scrubbed":21,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:57:15.601350+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0024472040000000001,"stat_sum":{"num_bytes":19,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":20,"num_read":10472,"num_read_kb":52916,"num_write":1098,"num_write_kb":37937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":649014,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"176'902","reported_seq":3584,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.962343+0000","last_change":"2024-04-23T21:24:15.962343+0000","last_active":"2024-04-23T21:24:15.962343+0000","last_peered":"2024-04-23T21:24:15.962343+0000","last_clean":"2024-04-23T21:24:15.962343+0000","last_became_active":"2024-04-23T21:23:54.204905+0000","last_became_peered":"2024-04-23T21:23:54.204905+0000","last_unstale":"2024-04-23T21:24:15.962343+0000","last_undegraded":"2024-04-23T21:24:15.962343+0000","last_fullsized":"2024-04-23T21:24:15.962343+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'902","last_scrub_stamp":"2024-04-23T21:24:15.962217+0000","last_deep_scrub":"176'902","last_deep_scrub_stamp":"2024-04-23T21:24:15.962217+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.962217+0000","objects_scrubbed":22,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:03:24.485298+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.000931546,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":5596,"num_read_kb":15018,"num_write":1147,"num_write_kb":9882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":7879700,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"176'763","reported_seq":6285,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:20.139574+0000","last_change":"2024-04-23T21:24:20.139640+0000","last_active":"2024-04-23T21:24:20.139574+0000","last_peered":"2024-04-23T21:24:20.139574+0000","last_clean":"2024-04-23T21:24:20.139574+0000","last_became_active":"2024-04-23T21:23:54.207451+0000","last_became_peered":"2024-04-23T21:23:54.207451+0000","last_unstale":"2024-04-23T21:24:20.139574+0000","last_undegraded":"2024-04-23T21:24:20.139574+0000","last_fullsized":"2024-04-23T21:24:20.139574+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'763","last_scrub_stamp":"2024-04-23T21:24:20.139485+0000","last_deep_scrub":"176'763","last_deep_scrub_stamp":"2024-04-23T21:24:20.139485+0000","last_clean_scrub_stamp":"2024-04-23T21:24:20.139485+0000","objects_scrubbed":21,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:54:04.719624+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.001551569,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":8100,"num_read_kb":25182,"num_write":879,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":6053113,"num_keys_recovered":22,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"176'640","reported_seq":3532,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.893288+0000","last_change":"2024-04-23T21:24:16.893315+0000","last_active":"2024-04-23T21:24:16.893288+0000","last_peered":"2024-04-23T21:24:16.893288+0000","last_clean":"2024-04-23T21:24:16.893288+0000","last_became_active":"2024-04-23T21:23:53.206644+0000","last_became_peered":"2024-04-23T21:23:53.206644+0000","last_unstale":"2024-04-23T21:24:16.893288+0000","last_undegraded":"2024-04-23T21:24:16.893288+0000","last_fullsized":"2024-04-23T21:24:16.893288+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'640","last_scrub_stamp":"2024-04-23T21:24:16.893206+0000","last_deep_scrub":"176'640","last_deep_scrub_stamp":"2024-04-23T21:24:16.893206+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.893206+0000","objects_scrubbed":28,"log_size":640,"log_dups_size":0,"ondisk_log_size":640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:37:59.700952+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0.0016571629999999999,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":4334,"num_read_kb":31182,"num_write":620,"num_write_kb":28227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":3057208,"num_keys_recovered":9,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"176'482","reported_seq":5357,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.691409+0000","last_change":"2024-04-23T21:24:15.691469+0000","last_active":"2024-04-23T21:24:15.691409+0000","last_peered":"2024-04-23T21:24:15.691409+0000","last_clean":"2024-04-23T21:24:15.691409+0000","last_became_active":"2024-04-23T21:23:50.305895+0000","last_became_peered":"2024-04-23T21:23:50.305895+0000","last_unstale":"2024-04-23T21:24:15.691409+0000","last_undegraded":"2024-04-23T21:24:15.691409+0000","last_fullsized":"2024-04-23T21:24:15.691409+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'482","last_scrub_stamp":"2024-04-23T21:24:15.691320+0000","last_deep_scrub":"176'482","last_deep_scrub_stamp":"2024-04-23T21:24:15.691320+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.691320+0000","objects_scrubbed":21,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:40:57.182235+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0016397670000000001,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":6184,"num_read_kb":36256,"num_write":472,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":1689997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"176'430","reported_seq":5206,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:22.230209+0000","last_change":"2024-04-23T21:24:22.230266+0000","last_active":"2024-04-23T21:24:22.230209+0000","last_peered":"2024-04-23T21:24:22.230209+0000","last_clean":"2024-04-23T21:24:22.230209+0000","last_became_active":"2024-04-23T21:23:52.210616+0000","last_became_peered":"2024-04-23T21:23:52.210616+0000","last_unstale":"2024-04-23T21:24:22.230209+0000","last_undegraded":"2024-04-23T21:24:22.230209+0000","last_fullsized":"2024-04-23T21:24:22.230209+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'430","last_scrub_stamp":"2024-04-23T21:24:22.230122+0000","last_deep_scrub":"176'430","last_deep_scrub_stamp":"2024-04-23T21:24:22.230122+0000","last_clean_scrub_stamp":"2024-04-23T21:24:22.230122+0000","objects_scrubbed":19,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:48:22.045055+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0066448410000000003,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":4558,"num_read_kb":26966,"num_write":373,"num_write_kb":9620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":6024624,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"176'715","reported_seq":6709,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:18.336804+0000","last_change":"2024-04-23T21:24:18.336860+0000","last_active":"2024-04-23T21:24:18.336804+0000","last_peered":"2024-04-23T21:24:18.336804+0000","last_clean":"2024-04-23T21:24:18.336804+0000","last_became_active":"2024-04-23T21:23:53.201566+0000","last_became_peered":"2024-04-23T21:23:53.201566+0000","last_unstale":"2024-04-23T21:24:18.336804+0000","last_undegraded":"2024-04-23T21:24:18.336804+0000","last_fullsized":"2024-04-23T21:24:18.336804+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'715","last_scrub_stamp":"2024-04-23T21:24:18.336718+0000","last_deep_scrub":"176'715","last_deep_scrub_stamp":"2024-04-23T21:24:18.336718+0000","last_clean_scrub_stamp":"2024-04-23T21:24:18.336718+0000","objects_scrubbed":22,"log_size":715,"log_dups_size":0,"ondisk_log_size":715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:46:22.317375+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.0072752829999999996,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":7880,"num_read_kb":52446,"num_write":809,"num_write_kb":17087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":210,"num_bytes_recovered":2103972,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":353,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.618224+0000","last_change":"2024-04-23T21:24:17.618249+0000","last_active":"2024-04-23T21:24:17.618224+0000","last_peered":"2024-04-23T21:24:17.618224+0000","last_clean":"2024-04-23T21:24:17.618224+0000","last_became_active":"2024-04-23T21:23:50.314877+0000","last_became_peered":"2024-04-23T21:23:50.314877+0000","last_unstale":"2024-04-23T21:24:17.618224+0000","last_undegraded":"2024-04-23T21:24:17.618224+0000","last_fullsized":"2024-04-23T21:24:17.618224+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:17.618125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:17.618125+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.618125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T00:06:00.504466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"176'502","reported_seq":3121,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:20.739630+0000","last_change":"2024-04-23T21:24:20.739674+0000","last_active":"2024-04-23T21:24:20.739630+0000","last_peered":"2024-04-23T21:24:20.739630+0000","last_clean":"2024-04-23T21:24:20.739630+0000","last_became_active":"2024-04-23T21:24:07.422375+0000","last_became_peered":"2024-04-23T21:24:07.422375+0000","last_unstale":"2024-04-23T21:24:20.739630+0000","last_undegraded":"2024-04-23T21:24:20.739630+0000","last_fullsized":"2024-04-23T21:24:20.739630+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'502","last_scrub_stamp":"2024-04-23T21:24:20.739527+0000","last_deep_scrub":"176'502","last_deep_scrub_stamp":"2024-04-23T21:24:20.739527+0000","last_clean_scrub_stamp":"2024-04-23T21:24:20.739527+0000","objects_scrubbed":13,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:36:08.641133+0000","scrub_duration":8,"objects_trimmed":22,"snaptrim_duration":0.079589640000000003,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":2675,"num_read_kb":19064,"num_write":531,"num_write_kb":12051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":12815775,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":103,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:18.570607+0000","last_change":"2024-04-23T21:24:18.570631+0000","last_active":"2024-04-23T21:24:18.570607+0000","last_peered":"2024-04-23T21:24:18.570607+0000","last_clean":"2024-04-23T21:24:18.570607+0000","last_became_active":"2024-04-23T21:23:51.306205+0000","last_became_peered":"2024-04-23T21:23:51.306205+0000","last_unstale":"2024-04-23T21:24:18.570607+0000","last_undegraded":"2024-04-23T21:24:18.570607+0000","last_fullsized":"2024-04-23T21:24:18.570607+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:18.570502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:18.570502+0000","last_clean_scrub_stamp":"2024-04-23T21:24:18.570502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:40:12.134515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"176'479","reported_seq":4659,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.004417+0000","last_change":"2024-04-23T21:24:15.004477+0000","last_active":"2024-04-23T21:24:15.004417+0000","last_peered":"2024-04-23T21:24:15.004417+0000","last_clean":"2024-04-23T21:24:15.004417+0000","last_became_active":"2024-04-23T21:21:52.959640+0000","last_became_peered":"2024-04-23T21:21:52.959640+0000","last_unstale":"2024-04-23T21:24:15.004417+0000","last_undegraded":"2024-04-23T21:24:15.004417+0000","last_fullsized":"2024-04-23T21:24:15.004417+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'479","last_scrub_stamp":"2024-04-23T21:24:15.004334+0000","last_deep_scrub":"176'479","last_deep_scrub_stamp":"2024-04-23T21:24:15.004334+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.004334+0000","objects_scrubbed":11,"log_size":479,"log_dups_size":0,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T21:34:33.942940+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.001786649,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":5347,"num_read_kb":35126,"num_write":383,"num_write_kb":17889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":543689,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":341,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.346147+0000","last_change":"2024-04-23T21:24:15.346184+0000","last_active":"2024-04-23T21:24:15.346147+0000","last_peered":"2024-04-23T21:24:15.346147+0000","last_clean":"2024-04-23T21:24:15.346147+0000","last_became_active":"2024-04-23T21:21:09.797672+0000","last_became_peered":"2024-04-23T21:21:09.797672+0000","last_unstale":"2024-04-23T21:24:15.346147+0000","last_undegraded":"2024-04-23T21:24:15.346147+0000","last_fullsized":"2024-04-23T21:24:15.346147+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:15.346051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:15.346051+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.346051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:15:51.566124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"176'463","reported_seq":3996,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.163219+0000","last_change":"2024-04-23T21:24:17.163271+0000","last_active":"2024-04-23T21:24:17.163219+0000","last_peered":"2024-04-23T21:24:17.163219+0000","last_clean":"2024-04-23T21:24:17.163219+0000","last_became_active":"2024-04-23T21:23:52.214611+0000","last_became_peered":"2024-04-23T21:23:52.214611+0000","last_unstale":"2024-04-23T21:24:17.163219+0000","last_undegraded":"2024-04-23T21:24:17.163219+0000","last_fullsized":"2024-04-23T21:24:17.163219+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'463","last_scrub_stamp":"2024-04-23T21:24:17.163128+0000","last_deep_scrub":"176'463","last_deep_scrub_stamp":"2024-04-23T21:24:17.163128+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.163128+0000","objects_scrubbed":25,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:43:12.922605+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.005602363,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":3812,"num_read_kb":24547,"num_write":487,"num_write_kb":11837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":287,"num_bytes_recovered":10975909,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":49,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:15.078085+0000","last_change":"2024-04-23T21:24:15.078119+0000","last_active":"2024-04-23T21:24:15.078085+0000","last_peered":"2024-04-23T21:24:15.078085+0000","last_clean":"2024-04-23T21:24:15.078085+0000","last_became_active":"2024-04-23T21:23:50.308711+0000","last_became_peered":"2024-04-23T21:23:50.308711+0000","last_unstale":"2024-04-23T21:24:15.078085+0000","last_undegraded":"2024-04-23T21:24:15.078085+0000","last_fullsized":"2024-04-23T21:24:15.078085+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:15.077975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:15.077975+0000","last_clean_scrub_stamp":"2024-04-23T21:24:15.077975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:05:22.849648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"176'453","reported_seq":4830,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.882829+0000","last_change":"2024-04-23T21:24:17.882861+0000","last_active":"2024-04-23T21:24:17.882829+0000","last_peered":"2024-04-23T21:24:17.882829+0000","last_clean":"2024-04-23T21:24:17.882829+0000","last_became_active":"2024-04-23T21:23:55.204776+0000","last_became_peered":"2024-04-23T21:23:55.204776+0000","last_unstale":"2024-04-23T21:24:17.882829+0000","last_undegraded":"2024-04-23T21:24:17.882829+0000","last_fullsized":"2024-04-23T21:24:17.882829+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'453","last_scrub_stamp":"2024-04-23T21:24:17.882739+0000","last_deep_scrub":"176'453","last_deep_scrub_stamp":"2024-04-23T21:24:17.882739+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.882739+0000","objects_scrubbed":26,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:47:22.400470+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001789896,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5969,"num_read_kb":33589,"num_write":360,"num_write_kb":18243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":1121086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"176'440","reported_seq":8803,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:13.870714+0000","last_change":"2024-04-23T21:24:13.870750+0000","last_active":"2024-04-23T21:24:13.870714+0000","last_peered":"2024-04-23T21:24:13.870714+0000","last_clean":"2024-04-23T21:24:13.870714+0000","last_became_active":"2024-04-23T21:23:53.203223+0000","last_became_peered":"2024-04-23T21:23:53.203223+0000","last_unstale":"2024-04-23T21:24:13.870714+0000","last_undegraded":"2024-04-23T21:24:13.870714+0000","last_fullsized":"2024-04-23T21:24:13.870714+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'440","last_scrub_stamp":"2024-04-23T21:24:13.870646+0000","last_deep_scrub":"176'440","last_deep_scrub_stamp":"2024-04-23T21:24:13.870646+0000","last_clean_scrub_stamp":"2024-04-23T21:24:13.870646+0000","objects_scrubbed":17,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:08:56.560826+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0023365700000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":8322,"num_read_kb":76065,"num_write":495,"num_write_kb":35593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"176'119","reported_seq":453,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.293140+0000","last_change":"2024-04-23T21:24:16.293178+0000","last_active":"2024-04-23T21:24:16.293140+0000","last_peered":"2024-04-23T21:24:16.293140+0000","last_clean":"2024-04-23T21:24:16.293140+0000","last_became_active":"2024-04-23T21:23:50.313636+0000","last_became_peered":"2024-04-23T21:23:50.313636+0000","last_unstale":"2024-04-23T21:24:16.293140+0000","last_undegraded":"2024-04-23T21:24:16.293140+0000","last_fullsized":"2024-04-23T21:24:16.293140+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'119","last_scrub_stamp":"2024-04-23T21:24:16.293047+0000","last_deep_scrub":"176'119","last_deep_scrub_stamp":"2024-04-23T21:24:16.293047+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.293047+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:14:59.859422+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":255,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.931988+0000","last_change":"2024-04-23T21:24:16.932028+0000","last_active":"2024-04-23T21:24:16.931988+0000","last_peered":"2024-04-23T21:24:16.931988+0000","last_clean":"2024-04-23T21:24:16.931988+0000","last_became_active":"2024-04-23T21:24:05.472904+0000","last_became_peered":"2024-04-23T21:24:05.472904+0000","last_unstale":"2024-04-23T21:24:16.931988+0000","last_undegraded":"2024-04-23T21:24:16.931988+0000","last_fullsized":"2024-04-23T21:24:16.931988+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:16.931899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:16.931899+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.931899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:45:54.336054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"176'357","reported_seq":1419,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:14.017359+0000","last_change":"2024-04-23T21:24:14.017429+0000","last_active":"2024-04-23T21:24:14.017359+0000","last_peered":"2024-04-23T21:24:14.017359+0000","last_clean":"2024-04-23T21:24:14.017359+0000","last_became_active":"2024-04-23T21:21:49.939093+0000","last_became_peered":"2024-04-23T21:21:49.939093+0000","last_unstale":"2024-04-23T21:24:14.017359+0000","last_undegraded":"2024-04-23T21:24:14.017359+0000","last_fullsized":"2024-04-23T21:24:14.017359+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'357","last_scrub_stamp":"2024-04-23T21:24:14.017274+0000","last_deep_scrub":"176'357","last_deep_scrub_stamp":"2024-04-23T21:24:14.017274+0000","last_clean_scrub_stamp":"2024-04-23T21:24:14.017274+0000","objects_scrubbed":20,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:54.744106+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001877734,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":3396,"num_read_kb":9711,"num_write":354,"num_write_kb":3482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":799251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":57,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:19.234505+0000","last_change":"2024-04-23T21:24:19.234547+0000","last_active":"2024-04-23T21:24:19.234505+0000","last_peered":"2024-04-23T21:24:19.234505+0000","last_clean":"2024-04-23T21:24:19.234505+0000","last_became_active":"2024-04-23T21:23:51.329021+0000","last_became_peered":"2024-04-23T21:23:51.329021+0000","last_unstale":"2024-04-23T21:24:19.234505+0000","last_undegraded":"2024-04-23T21:24:19.234505+0000","last_fullsized":"2024-04-23T21:24:19.234505+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:19.234401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:19.234401+0000","last_clean_scrub_stamp":"2024-04-23T21:24:19.234401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:47:06.736697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"176'987","reported_seq":9541,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:19.349092+0000","last_change":"2024-04-23T21:24:19.349142+0000","last_active":"2024-04-23T21:24:19.349092+0000","last_peered":"2024-04-23T21:24:19.349092+0000","last_clean":"2024-04-23T21:24:19.349092+0000","last_became_active":"2024-04-23T21:23:54.212643+0000","last_became_peered":"2024-04-23T21:23:54.212643+0000","last_unstale":"2024-04-23T21:24:19.349092+0000","last_undegraded":"2024-04-23T21:24:19.349092+0000","last_fullsized":"2024-04-23T21:24:19.349092+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'987","last_scrub_stamp":"2024-04-23T21:24:19.348999+0000","last_deep_scrub":"176'987","last_deep_scrub_stamp":"2024-04-23T21:24:19.348999+0000","last_clean_scrub_stamp":"2024-04-23T21:24:19.348999+0000","objects_scrubbed":19,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:40:16.212300+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0057000330000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":10728,"num_read_kb":58181,"num_write":1175,"num_write_kb":33551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":15013851,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":194,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:18.718400+0000","last_change":"2024-04-23T21:24:18.718425+0000","last_active":"2024-04-23T21:24:18.718400+0000","last_peered":"2024-04-23T21:24:18.718400+0000","last_clean":"2024-04-23T21:24:18.718400+0000","last_became_active":"2024-04-23T21:24:07.418557+0000","last_became_peered":"2024-04-23T21:24:07.418557+0000","last_unstale":"2024-04-23T21:24:18.718400+0000","last_undegraded":"2024-04-23T21:24:18.718400+0000","last_fullsized":"2024-04-23T21:24:18.718400+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:18.718310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:18.718310+0000","last_clean_scrub_stamp":"2024-04-23T21:24:18.718310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:57:21.541597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"176'653","reported_seq":5588,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:18.165081+0000","last_change":"2024-04-23T21:24:18.165160+0000","last_active":"2024-04-23T21:24:18.165081+0000","last_peered":"2024-04-23T21:24:18.165081+0000","last_clean":"2024-04-23T21:24:18.165081+0000","last_became_active":"2024-04-23T21:23:53.206386+0000","last_became_peered":"2024-04-23T21:23:53.206386+0000","last_unstale":"2024-04-23T21:24:18.165081+0000","last_undegraded":"2024-04-23T21:24:18.165081+0000","last_fullsized":"2024-04-23T21:24:18.165081+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'653","last_scrub_stamp":"2024-04-23T21:24:18.164975+0000","last_deep_scrub":"176'653","last_deep_scrub_stamp":"2024-04-23T21:24:18.164975+0000","last_clean_scrub_stamp":"2024-04-23T21:24:18.164975+0000","objects_scrubbed":17,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:54:00.732207+0000","scrub_duration":12,"objects_trimmed":10,"snaptrim_duration":0.070001033000000004,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7860,"num_read_kb":15083,"num_write":763,"num_write_kb":8876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":6020779,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:18.221306+0000","last_change":"2024-04-23T21:24:18.221331+0000","last_active":"2024-04-23T21:24:18.221306+0000","last_peered":"2024-04-23T21:24:18.221306+0000","last_clean":"2024-04-23T21:24:18.221306+0000","last_became_active":"2024-04-23T21:23:51.327847+0000","last_became_peered":"2024-04-23T21:23:51.327847+0000","last_unstale":"2024-04-23T21:24:18.221306+0000","last_undegraded":"2024-04-23T21:24:18.221306+0000","last_fullsized":"2024-04-23T21:24:18.221306+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T21:24:18.221208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T21:24:18.221208+0000","last_clean_scrub_stamp":"2024-04-23T21:24:18.221208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:24:03.599067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"176'336","reported_seq":3279,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.113886+0000","last_change":"2024-04-23T21:24:16.113926+0000","last_active":"2024-04-23T21:24:16.113886+0000","last_peered":"2024-04-23T21:24:16.113886+0000","last_clean":"2024-04-23T21:24:16.113886+0000","last_became_active":"2024-04-23T21:23:52.210319+0000","last_became_peered":"2024-04-23T21:23:52.210319+0000","last_unstale":"2024-04-23T21:24:16.113886+0000","last_undegraded":"2024-04-23T21:24:16.113886+0000","last_fullsized":"2024-04-23T21:24:16.113886+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'336","last_scrub_stamp":"2024-04-23T21:24:16.113817+0000","last_deep_scrub":"176'336","last_deep_scrub_stamp":"2024-04-23T21:24:16.113817+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.113817+0000","objects_scrubbed":19,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T04:07:15.033676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001730243,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":2342,"num_read_kb":15570,"num_write":288,"num_write_kb":12012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":13873454,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"176'385","reported_seq":7711,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.375413+0000","last_change":"2024-04-23T21:24:16.375457+0000","last_active":"2024-04-23T21:24:16.375413+0000","last_peered":"2024-04-23T21:24:16.375413+0000","last_clean":"2024-04-23T21:24:16.375413+0000","last_became_active":"2024-04-23T21:21:17.779617+0000","last_became_peered":"2024-04-23T21:21:17.779617+0000","last_unstale":"2024-04-23T21:24:16.375413+0000","last_undegraded":"2024-04-23T21:24:16.375413+0000","last_fullsized":"2024-04-23T21:24:16.375413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'385","last_scrub_stamp":"2024-04-23T21:24:16.375331+0000","last_deep_scrub":"176'385","last_deep_scrub_stamp":"2024-04-23T21:24:16.375331+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.375331+0000","objects_scrubbed":17,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T01:18:10.353068+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.004485897,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":7064,"num_read_kb":46747,"num_write":361,"num_write_kb":15724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"176'643","reported_seq":4337,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:16.605733+0000","last_change":"2024-04-23T21:24:16.605852+0000","last_active":"2024-04-23T21:24:16.605733+0000","last_peered":"2024-04-23T21:24:16.605733+0000","last_clean":"2024-04-23T21:24:16.605733+0000","last_became_active":"2024-04-23T21:21:17.781614+0000","last_became_peered":"2024-04-23T21:21:17.781614+0000","last_unstale":"2024-04-23T21:24:16.605733+0000","last_undegraded":"2024-04-23T21:24:16.605733+0000","last_fullsized":"2024-04-23T21:24:16.605733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'643","last_scrub_stamp":"2024-04-23T21:24:16.605628+0000","last_deep_scrub":"176'643","last_deep_scrub_stamp":"2024-04-23T21:24:16.605628+0000","last_clean_scrub_stamp":"2024-04-23T21:24:16.605628+0000","objects_scrubbed":22,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:01:55.477375+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00088070399999999995,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6272,"num_read_kb":13530,"num_write":577,"num_write_kb":14999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"176'512","reported_seq":3037,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:21.143905+0000","last_change":"2024-04-23T21:24:21.143958+0000","last_active":"2024-04-23T21:24:21.143905+0000","last_peered":"2024-04-23T21:24:21.143905+0000","last_clean":"2024-04-23T21:24:21.143905+0000","last_became_active":"2024-04-23T21:23:54.207552+0000","last_became_peered":"2024-04-23T21:23:54.207552+0000","last_unstale":"2024-04-23T21:24:21.143905+0000","last_undegraded":"2024-04-23T21:24:21.143905+0000","last_fullsized":"2024-04-23T21:24:21.143905+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'512","last_scrub_stamp":"2024-04-23T21:24:21.143815+0000","last_deep_scrub":"176'512","last_deep_scrub_stamp":"2024-04-23T21:24:21.143815+0000","last_clean_scrub_stamp":"2024-04-23T21:24:21.143815+0000","objects_scrubbed":16,"log_size":512,"log_dups_size":0,"ondisk_log_size":512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T07:05:42.409129+0000","scrub_duration":16,"objects_trimmed":2,"snaptrim_duration":0.0020775120000000001,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":2826,"num_read_kb":26002,"num_write":499,"num_write_kb":14318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":10273352,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"176'455","reported_seq":2197,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:19.712859+0000","last_change":"2024-04-23T21:24:19.712911+0000","last_active":"2024-04-23T21:24:19.712859+0000","last_peered":"2024-04-23T21:24:19.712859+0000","last_clean":"2024-04-23T21:24:19.712859+0000","last_became_active":"2024-04-23T21:24:07.422312+0000","last_became_peered":"2024-04-23T21:24:07.422312+0000","last_unstale":"2024-04-23T21:24:19.712859+0000","last_undegraded":"2024-04-23T21:24:19.712859+0000","last_fullsized":"2024-04-23T21:24:19.712859+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'455","last_scrub_stamp":"2024-04-23T21:24:19.712769+0000","last_deep_scrub":"176'455","last_deep_scrub_stamp":"2024-04-23T21:24:19.712769+0000","last_clean_scrub_stamp":"2024-04-23T21:24:19.712769+0000","objects_scrubbed":18,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T02:43:02.069347+0000","scrub_duration":13,"objects_trimmed":24,"snaptrim_duration":0.075475576000000003,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":1674,"num_read_kb":18858,"num_write":375,"num_write_kb":17741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":9926018,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"176'342","reported_seq":3158,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:17.281849+0000","last_change":"2024-04-23T21:24:17.281910+0000","last_active":"2024-04-23T21:24:17.281849+0000","last_peered":"2024-04-23T21:24:17.281849+0000","last_clean":"2024-04-23T21:24:17.281849+0000","last_became_active":"2024-04-23T21:23:50.306096+0000","last_became_peered":"2024-04-23T21:23:50.306096+0000","last_unstale":"2024-04-23T21:24:17.281849+0000","last_undegraded":"2024-04-23T21:24:17.281849+0000","last_fullsized":"2024-04-23T21:24:17.281849+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'342","last_scrub_stamp":"2024-04-23T21:24:17.281758+0000","last_deep_scrub":"176'342","last_deep_scrub_stamp":"2024-04-23T21:24:17.281758+0000","last_clean_scrub_stamp":"2024-04-23T21:24:17.281758+0000","objects_scrubbed":22,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T22:28:59.238193+0000","scrub_duration":14,"objects_trimmed":1,"snaptrim_duration":0.0059788380000000002,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":2720,"num_read_kb":11635,"num_write":280,"num_write_kb":11074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":1304512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"176'653","reported_seq":2854,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:23.235088+0000","last_change":"2024-04-23T21:24:23.235163+0000","last_active":"2024-04-23T21:24:23.235088+0000","last_peered":"2024-04-23T21:24:23.235088+0000","last_clean":"2024-04-23T21:24:23.235088+0000","last_became_active":"2024-04-23T21:23:53.200384+0000","last_became_peered":"2024-04-23T21:23:53.200384+0000","last_unstale":"2024-04-23T21:24:23.235088+0000","last_undegraded":"2024-04-23T21:24:23.235088+0000","last_fullsized":"2024-04-23T21:24:23.235088+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'653","last_scrub_stamp":"2024-04-23T21:24:23.234974+0000","last_deep_scrub":"176'653","last_deep_scrub_stamp":"2024-04-23T21:24:23.234974+0000","last_clean_scrub_stamp":"2024-04-23T21:24:23.234974+0000","objects_scrubbed":18,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:10:21.754189+0000","scrub_duration":15,"objects_trimmed":2,"snaptrim_duration":0.010127324,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4051,"num_read_kb":11717,"num_write":728,"num_write_kb":15970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":28116,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"176'1095","reported_seq":11132,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:24.219007+0000","last_change":"2024-04-23T21:24:24.219061+0000","last_active":"2024-04-23T21:24:24.219007+0000","last_peered":"2024-04-23T21:24:24.219007+0000","last_clean":"2024-04-23T21:24:24.219007+0000","last_became_active":"2024-04-23T21:23:53.201061+0000","last_became_peered":"2024-04-23T21:23:53.201061+0000","last_unstale":"2024-04-23T21:24:24.219007+0000","last_undegraded":"2024-04-23T21:24:24.219007+0000","last_fullsized":"2024-04-23T21:24:24.219007+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'1095","last_scrub_stamp":"2024-04-23T21:24:24.218903+0000","last_deep_scrub":"176'1095","last_deep_scrub_stamp":"2024-04-23T21:24:24.218903+0000","last_clean_scrub_stamp":"2024-04-23T21:24:24.218903+0000","objects_scrubbed":17,"log_size":1095,"log_dups_size":0,"ondisk_log_size":1095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:38.596219+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0034327860000000002,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":14047,"num_read_kb":59481,"num_write":1400,"num_write_kb":16442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":459771,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"176'567","reported_seq":3893,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:19.620115+0000","last_change":"2024-04-23T21:24:19.620230+0000","last_active":"2024-04-23T21:24:19.620115+0000","last_peered":"2024-04-23T21:24:19.620115+0000","last_clean":"2024-04-23T21:24:19.620115+0000","last_became_active":"2024-04-23T21:23:52.202920+0000","last_became_peered":"2024-04-23T21:23:52.202920+0000","last_unstale":"2024-04-23T21:24:19.620115+0000","last_undegraded":"2024-04-23T21:24:19.620115+0000","last_fullsized":"2024-04-23T21:24:19.620115+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'567","last_scrub_stamp":"2024-04-23T21:24:19.620011+0000","last_deep_scrub":"176'567","last_deep_scrub_stamp":"2024-04-23T21:24:19.620011+0000","last_clean_scrub_stamp":"2024-04-23T21:24:19.620011+0000","objects_scrubbed":18,"log_size":567,"log_dups_size":0,"ondisk_log_size":567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T05:59:29.945285+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00088461599999999996,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4693,"num_read_kb":33996,"num_write":568,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"176'320","reported_seq":2861,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:20.254785+0000","last_change":"2024-04-23T21:24:20.254846+0000","last_active":"2024-04-23T21:24:20.254785+0000","last_peered":"2024-04-23T21:24:20.254785+0000","last_clean":"2024-04-23T21:24:20.254785+0000","last_became_active":"2024-04-23T21:23:52.203063+0000","last_became_peered":"2024-04-23T21:23:52.203063+0000","last_unstale":"2024-04-23T21:24:20.254785+0000","last_undegraded":"2024-04-23T21:24:20.254785+0000","last_fullsized":"2024-04-23T21:24:20.254785+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'320","last_scrub_stamp":"2024-04-23T21:24:20.254662+0000","last_deep_scrub":"176'320","last_deep_scrub_stamp":"2024-04-23T21:24:20.254662+0000","last_clean_scrub_stamp":"2024-04-23T21:24:20.254662+0000","objects_scrubbed":11,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:49:59.389394+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0013287049999999999,"stat_sum":{"num_bytes":0,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":11,"num_read":1900,"num_read_kb":20604,"num_write":305,"num_write_kb":14401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":508615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"176'975","reported_seq":4184,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:14.924975+0000","last_change":"2024-04-23T21:24:14.925035+0000","last_active":"2024-04-23T21:24:14.924975+0000","last_peered":"2024-04-23T21:24:14.924975+0000","last_clean":"2024-04-23T21:24:14.924975+0000","last_became_active":"2024-04-23T21:23:53.203295+0000","last_became_peered":"2024-04-23T21:23:53.203295+0000","last_unstale":"2024-04-23T21:24:14.924975+0000","last_undegraded":"2024-04-23T21:24:14.924975+0000","last_fullsized":"2024-04-23T21:24:14.924975+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'975","last_scrub_stamp":"2024-04-23T21:24:14.924892+0000","last_deep_scrub":"176'975","last_deep_scrub_stamp":"2024-04-23T21:24:14.924892+0000","last_clean_scrub_stamp":"2024-04-23T21:24:14.924892+0000","objects_scrubbed":20,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T23:40:32.854105+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0014334300000000001,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":6073,"num_read_kb":40110,"num_write":1213,"num_write_kb":21666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":1369323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"176'944","reported_seq":3551,"reported_epoch":176,"state":"active+clean","last_fresh":"2024-04-23T21:24:21.235301+0000","last_change":"2024-04-23T21:24:21.235352+0000","last_active":"2024-04-23T21:24:21.235301+0000","last_peered":"2024-04-23T21:24:21.235301+0000","last_clean":"2024-04-23T21:24:21.235301+0000","last_became_active":"2024-04-23T21:23:52.207485+0000","last_became_peered":"2024-04-23T21:23:52.207485+0000","last_unstale":"2024-04-23T21:24:21.235301+0000","last_undegraded":"2024-04-23T21:24:21.235301+0000","last_fullsized":"2024-04-23T21:24:21.235301+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'944","last_scrub_stamp":"2024-04-23T21:24:21.235215+0000","last_deep_scrub":"176'944","last_deep_scrub_stamp":"2024-04-23T21:24:21.235215+0000","last_clean_scrub_stamp":"2024-04-23T21:24:21.235215+0000","objects_scrubbed":19,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T03:59:56.416584+0000","scrub_duration":15,"objects_trimmed":4,"snaptrim_duration":0.000999661,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":5526,"num_read_kb":39400,"num_write":1328,"num_write_kb":24127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":548982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":620,"num_object_clones":0,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":616,"num_read":191132,"num_read_kb":1022770,"num_write":20201,"num_write_kb":598261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5203,"num_bytes_recovered":138643378,"num_keys_recovered":304,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},"log_size":18841,"ondisk_log_size":18841,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":66480,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":2350,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47200,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324640,"statfs":{"total":96636764160,"available":96588431360,"internally_reserved":0,"allocated":2187264,"data_stored":1631096,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574893,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29088,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342752,"statfs":{"total":96636764160,"available":96606978048,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":157,"seq":674309865483,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35104,"kb_used_data":2072,"kb_used_omap":31,"kb_used_meta":32992,"kb_avail":94336736,"statfs":{"total":96636764160,"available":96600817664,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32064,"internal_metadata":33784512},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73508,"kb_used_data":2076,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298332,"statfs":{"total":96636764160,"available":96561491968,"internally_reserved":0,"allocated":2125824,"data_stored":1040747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574893,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55456,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316384,"statfs":{"total":96636764160,"available":96579977216,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41568,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330272,"statfs":{"total":96636764160,"available":96594198528,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":45,"seq":193273528355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34644,"kb_used_data":2124,"kb_used_omap":6,"kb_used_meta":32505,"kb_avail":94337196,"statfs":{"total":96636764160,"available":96601288704,"internally_reserved":0,"allocated":2174976,"data_stored":1631115,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":6748,"internal_metadata":33285540},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59616,"kb_used_data":2072,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312224,"statfs":{"total":96636764160,"available":96575717376,"internally_reserved":0,"allocated":2121728,"data_stored":1040728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34058,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":2350,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32422,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16106,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T21:24:36.799 DEBUG:teuthology.orchestra.run.smithi111:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-23T21:24:37.100 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-23T21:24:37.100 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-23T21:24:37.100 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-23T21:24:37.100 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.135 INFO:tasks.ceph.osd.0:Stopped 2024-04-23T21:24:37.135 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-23T21:24:37.135 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.209 INFO:tasks.ceph.osd.1:Stopped 2024-04-23T21:24:37.209 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-23T21:24:37.209 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.294 INFO:tasks.ceph.osd.2:Stopped 2024-04-23T21:24:37.294 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-23T21:24:37.294 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.341 INFO:tasks.ceph.osd.3:Stopped 2024-04-23T21:24:37.341 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-23T21:24:37.341 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.379 INFO:tasks.ceph.osd.4:Stopped 2024-04-23T21:24:37.379 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-23T21:24:37.379 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.453 INFO:tasks.ceph.osd.5:Stopped 2024-04-23T21:24:37.454 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-23T21:24:37.454 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.525 INFO:tasks.ceph.osd.6:Stopped 2024-04-23T21:24:37.526 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-23T21:24:37.526 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.603 INFO:tasks.ceph.osd.7:Stopped 2024-04-23T21:24:37.603 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-23T21:24:37.604 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-23T21:24:37.604 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.627 INFO:tasks.ceph.mgr.y:Stopped 2024-04-23T21:24:37.628 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-23T21:24:37.628 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.672 INFO:tasks.ceph.mgr.x:Stopped 2024-04-23T21:24:37.672 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-23T21:24:37.672 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-23T21:24:37.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.683 INFO:tasks.ceph.mon.a:Stopped 2024-04-23T21:24:37.683 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-23T21:24:37.683 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.734 INFO:tasks.ceph.mon.c:Stopped 2024-04-23T21:24:37.734 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-23T21:24:37.734 INFO:teuthology.orchestra.run:waiting for 300 2024-04-23T21:24:37.746 INFO:tasks.ceph.mon.b:Stopped 2024-04-23T21:24:37.747 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-23T21:24:37.747 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-04-23T21:24:37.808 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:24:37.809 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-23T21:24:38.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:24:38.087 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-23T21:24:38.152 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:24:38.152 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-23T21:24:38.201 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:24:38.201 DEBUG:teuthology.orchestra.run.smithi111:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-23T21:24:38.252 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:24:38.252 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-23T21:24:38.451 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:24:38.451 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-23T21:24:38.518 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:24:38.518 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-23T21:24:38.577 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:24:38.577 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-23T21:24:38.629 INFO:tasks.ceph:Archiving mon data... 2024-04-23T21:24:38.630 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/data/mon.a.tgz 2024-04-23T21:24:38.631 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-04-23T21:24:38.646 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.O5B6WRgkQw 2024-04-23T21:24:38.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.O5B6WRgkQw 2024-04-23T21:24:38.825 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.O5B6WRgkQw 2024-04-23T21:24:38.907 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.O5B6WRgkQw is 409KB 2024-04-23T21:24:39.004 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.O5B6WRgkQw 2024-04-23T21:24:39.019 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/data/mon.c.tgz 2024-04-23T21:24:39.019 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-04-23T21:24:39.073 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.FzZJvnYaL4 2024-04-23T21:24:39.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.FzZJvnYaL4 2024-04-23T21:24:39.245 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.FzZJvnYaL4 2024-04-23T21:24:39.325 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.FzZJvnYaL4 is 555KB 2024-04-23T21:24:39.423 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.FzZJvnYaL4 2024-04-23T21:24:39.438 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/data/mon.b.tgz 2024-04-23T21:24:39.438 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-04-23T21:24:39.456 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.TSvT4SGzGU 2024-04-23T21:24:39.456 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.TSvT4SGzGU 2024-04-23T21:24:39.647 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.TSvT4SGzGU 2024-04-23T21:24:39.730 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.TSvT4SGzGU is 555KB 2024-04-23T21:24:39.829 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.TSvT4SGzGU 2024-04-23T21:24:39.843 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-23T21:24:39.843 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T21:24:39.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-23T21:24:39.922 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T21:24:39.961 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-23T21:24:39.961 INFO:tasks.ceph:Archiving crash dumps... 2024-04-23T21:24:39.963 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi111/crash 2024-04-23T21:24:39.963 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T21:24:39.987 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi192/crash 2024-04-23T21:24:39.988 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-23T21:24:40.023 INFO:tasks.ceph:Compressing logs... 2024-04-23T21:24:40.023 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T21:24:40.032 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T21:24:40.053 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48178.log 2024-04-23T21:24:40.053 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44001.log 2024-04-23T21:24:40.109 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T21:24:40.109 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32133.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32213.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.32133.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32133.log.gz 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.32213.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32213.log.gz 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.5.log: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.42243.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32293.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/objectstore_tool.42243.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.42454.log 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.32293.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 0.0% 2024-04-23T21:24:40.110 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32293.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/objectstore_tool.42454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32002.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.32002.log: gzip -5 29.2% -- replaced with /var/log/ceph/ceph-client.admin.32002.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/objectstore_tool.42665.log 94.7% -- replaced with /var/log/ceph/objectstore_tool.42243.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32087.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/objectstore_tool.42665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32173.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.42454.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.32087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32253.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0%/var/log/ceph/ceph-client.admin.32173.log: -- replaced with /var/log/ceph/ceph-client.admin.32087.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32173.log.gz 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.43602.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-client.admin.32253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32253.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-04-23T21:24:40.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/objectstore_tool.43602.log: 94.9% -- replaced with /var/log/ceph/objectstore_tool.42665.log.gz 2024-04-23T21:24:40.112 INFO:teuthology.orchestra.run.smithi192.stderr: 94.8% -- replaced with /var/log/ceph/objectstore_tool.43602.log.gz 2024-04-23T21:24:40.112 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32042.log 2024-04-23T21:24:40.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.32042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32042.log.gz 2024-04-23T21:24:40.167 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-23T21:24:40.199 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48178.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46964.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44001.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35835.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46964.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56288.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59700.log 2024-04-23T21:24:40.203 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35835.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35891.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56288.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37591.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59700.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36242.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35891.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60402.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37591.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.36242.log: /var/log/ceph/ceph-client.admin.36525.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36242.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50134.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60402.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52143.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36525.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56047.log 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50134.log.gz 2024-04-23T21:24:40.204 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55114.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52143.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55770.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56047.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46802.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55114.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32351.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55770.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57733.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46802.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46521.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.32351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32351.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36240.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57733.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36568.log 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46521.log.gz 2024-04-23T21:24:40.205 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60612.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36240.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57059.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37327.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60612.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57059.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47210.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47210.log.gz 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44386.log 2024-04-23T21:24:40.206 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41297.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44386.log: /var/log/ceph/ceph-client.admin.44771.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44386.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42342.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41297.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44771.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57877.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42342.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58134.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46623.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57877.log.gz 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54999.log 2024-04-23T21:24:40.207 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58134.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46623.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 26.5% -- replaced with /var/log/ceph/ceph-client.admin.46623.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56686.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54999.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51151.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52995.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56686.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40490.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51151.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58980.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52995.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51928.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40490.log.gz 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60024.log 2024-04-23T21:24:40.208 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58980.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42907.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51928.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44204.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60024.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42907.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58978.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44204.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.31945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43826.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58978.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58978.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42165.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43826.log.gz 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45196.log 2024-04-23T21:24:40.209 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59540.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42165.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45196.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59636.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42610.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59540.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58806.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59636.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42610.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56490.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60192.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58806.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56490.log.gz 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44871.log 2024-04-23T21:24:40.210 INFO:teuthology.orchestra.run.smithi111.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36601.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60192.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44871.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48308.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36601.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60738.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57775.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48308.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55473.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60738.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49433.log 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57775.log.gz 2024-04-23T21:24:40.211 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59520.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55473.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49433.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49433.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60444.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59520.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58979.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41615.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60444.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58979.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41615.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53118.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53118.log.gz 2024-04-23T21:24:40.212 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52912.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60066.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47061.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52912.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60066.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.60066.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39245.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47061.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45337.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42240.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.39245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48527.log 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39245.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42240.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.57961.log -- replaced with /var/log/ceph/ceph-client.admin.42240.log.gz 2024-04-23T21:24:40.213 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48891.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48527.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57961.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37522.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60150.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48891.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.0.37522.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26127.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60150.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.60150.log.gz -5 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56625.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.0.37522.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph.tmp-client.admin.26127.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26127.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36019.log 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gz 2024-04-23T21:24:40.214 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56625.log.gz 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56211.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41018.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36019.log.gz 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56211.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.56211.log.gz --verbose 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.59244.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58386.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41018.log.gz 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59244.log.gz 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58386.log.gz 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60904.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gzgzip 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50580.log 2024-04-23T21:24:40.215 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60904.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54443.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50580.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60108.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54443.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60108.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58764.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40327.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45719.log 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58764.log.gz 2024-04-23T21:24:40.216 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38232.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45719.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45719.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: /var/log/ceph/ceph-client.admin.40687.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/objectstore_tool.38232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54363.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40687.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40687.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54363.log: /var/log/ceph/ceph-client.admin.45802.log: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54363.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45012.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58722.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.58260.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45012.log.gz 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58260.log: gzip 0.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-04-23T21:24:40.217 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36922.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5/var/log/ceph/ceph-client.admin.37201.log: --verbose -- /var/log/ceph/ceph-client.admin.60486.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz/var/log/ceph/ceph-client.admin.36922.log: 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58988.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36922.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36527.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60486.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36061.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58988.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54580.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36527.log: /var/log/ceph/ceph-client.admin.36061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51547.log 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36527.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36061.log.gz 2024-04-23T21:24:40.218 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54580.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54580.log.gz -5 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50518.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5/var/log/ceph/ceph-client.admin.51547.log: --verbose -- /var/log/ceph/ceph-client.admin.45678.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51547.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50518.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50319.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45678.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45678.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr: --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr: 95.0% -- replaced with /var/log/ceph/objectstore_tool.38232.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50319.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51191.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip/var/log/ceph/ceph-mon.c.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47701.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51191.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48605.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47701.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47701.log.gz 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.48444.log 2024-04-23T21:24:40.219 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48605.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48444.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48444.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr: -- /var/log/ceph/ceph-client.admin.54228.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39029.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54228.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52224.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/objectstore_tool.39029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54682.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52224.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60570.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52224.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40050.log 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.54682.log: /var/log/ceph/ceph-client.admin.60570.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54682.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60570.log.gz 2024-04-23T21:24:40.220 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53890.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40050.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49933.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53890.log.gz/var/log/ceph/ceph-client.admin.49933.log: 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49933.log.gz 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57578.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59283.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57578.log.gz 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr: 94.9% -- replaced with /var/log/ceph/objectstore_tool.39029.log.gz 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47922.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59283.log: 28.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59283.log.gz 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49699.log 2024-04-23T21:24:40.221 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42517.log: /var/log/ceph/ceph-client.admin.47922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47922.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45295.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49699.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44085.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49699.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50035.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45295.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58045.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44085.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50035.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41711.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55553.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58045.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41711.log.gz 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49632.log 2024-04-23T21:24:40.222 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49712.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49632.log: /var/log/ceph/ceph-client.admin.55553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55553.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49632.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58176.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49712.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36230.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49712.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53572.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58176.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36230.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59284.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53572.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50440.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40923.log 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59284.log: /var/log/ceph/ceph-client.admin.50440.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59284.log.gz 2024-04-23T21:24:40.223 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50440.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36243.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40923.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36243.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45941.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48179.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44768.log: /var/log/ceph/ceph-client.admin.45941.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45941.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36964.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52420.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48179.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36964.log.gz 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58344.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37243.log 2024-04-23T21:24:40.224 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58344.log: /var/log/ceph/ceph-client.admin.52420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52420.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58344.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36923.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37243.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58428.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36923.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.43941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45114.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58428.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46226.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54797.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45114.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46226.log.gz 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60234.log 2024-04-23T21:24:40.225 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51470.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60234.log: /var/log/ceph/ceph-client.admin.54797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54797.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60234.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59640.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36513.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59640.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36238.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36241.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36513.log: /var/log/ceph/ceph-client.admin.36238.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36513.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36238.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48021.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.37687.log 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36241.log.gz 2024-04-23T21:24:40.226 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48021.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50903.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55885.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50903.log: /var/log/ceph/objectstore_tool.37687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50903.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44570.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55885.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47962.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44570.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr: 94.5% -- replaced with /var/log/ceph/objectstore_tool.37687.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56210.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47962.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54026.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47962.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59351.log 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56210.log: /var/log/ceph/ceph-client.admin.54026.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56210.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54026.log.gz 2024-04-23T21:24:40.227 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53356.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53687.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59351.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53356.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45251.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53356.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57003.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53687.log: /var/log/ceph/ceph-client.admin.45251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53687.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45251.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36921.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36528.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57003.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36921.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45849.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59235.log 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45849.log: /var/log/ceph/ceph-client.admin.36528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36528.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45849.log.gz 2024-04-23T21:24:40.228 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58680.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60696.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59235.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58680.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47838.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49514.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60696.log: /var/log/ceph/ceph-client.admin.47838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47838.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60696.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37285.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49514.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60528.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49514.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56942.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37285.log: /var/log/ceph/ceph-client.admin.60528.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37285.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60528.log.gz 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59662.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58638.log 2024-04-23T21:24:40.229 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.56942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56942.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59662.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59662.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.47141.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52657.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58638.log: /var/log/ceph/ceph-client.admin.47141.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58638.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47141.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36823.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52657.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52657.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.58982.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36823.log: /var/log/ceph/ceph-client.admin.58982.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36823.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58982.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42863.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50742.log 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2024-04-23T21:24:40.230 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42863.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60864.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60864.log: /var/log/ceph/ceph-client.admin.50742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50742.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60864.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.37960.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49995.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46384.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49995.log: /var/log/ceph/objectstore_tool.37960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49995.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35977.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46384.log.gz 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32248.log 2024-04-23T21:24:40.231 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35977.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr: 94.7% -- replaced with /var/log/ceph/objectstore_tool.37960.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58470.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.32248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32248.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60822.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44550.log: /var/log/ceph/ceph-client.admin.58470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58470.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59348.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60822.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59348.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59348.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58987.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48486.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41836.log: /var/log/ceph/ceph-client.admin.58987.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58987.log.gz 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58092.log 2024-04-23T21:24:40.232 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41638.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48486.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58092.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44384.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58986.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41638.log: /var/log/ceph/ceph-client.admin.44384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44384.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41638.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58302.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58986.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55295.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58986.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58302.log: /var/log/ceph/ceph-client.admin.55295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55295.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58302.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46185.log 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-04-23T21:24:40.233 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36231.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36529.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36231.log: /var/log/ceph/ceph-client.admin.46185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36231.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46185.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53275.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36529.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53849.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36529.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59760.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.53849.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53849.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53275.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49092.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59760.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51826.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59760.log.gz 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.49092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50815.log 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49092.log.gz 0.0% 2024-04-23T21:24:40.234 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51826.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44649.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50815.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.44649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44649.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60318.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47442.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41897.log: /var/log/ceph/ceph-client.admin.60318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60318.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51109.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60654.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47442.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50560.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51109.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60654.log.gz 2024-04-23T21:24:40.235 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50560.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45581.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41338.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60276.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59600.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60276.log: /var/log/ceph/ceph-client.admin.41338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60276.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47499.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59600.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59600.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57300.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.47499.log: /var/log/ceph/ceph-client.admin.40978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47499.log.gz 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48871.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57300.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-04-23T21:24:40.236 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57300.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57396.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48871.log: /var/log/ceph/ceph-client.admin.59856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48871.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35933.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.57396.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51609.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57396.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.35933.log: /var/log/ceph/ceph-client.admin.51609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51609.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35933.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58554.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.45761.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49132.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52515.log 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58554.log: /var/log/ceph/ceph-client.admin.49132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49132.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58554.log.gz 2024-04-23T21:24:40.237 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36235.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52515.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57919.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52832.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36235.log: /var/log/ceph/ceph-client.admin.57919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36235.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57919.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42023.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.52832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52832.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36920.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41752.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.42023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58512.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42023.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41752.log.gz 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58003.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59898.log 2024-04-23T21:24:40.238 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58512.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58003.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51067.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60780.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.51067.log: /var/log/ceph/ceph-client.admin.59898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59898.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51067.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55316.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60780.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.55316.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55316.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.41574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59238.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37411.log 2024-04-23T21:24:40.239 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59238.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.40272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56128.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36917.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.37411.log: /var/log/ceph/ceph-client.admin.56128.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37411.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56128.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60360.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58985.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.36917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36917.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.60360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60360.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59243.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58596.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.59243.log: /var/log/ceph/ceph-client.admin.58985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58985.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59243.log.gz 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50114.log 2024-04-23T21:24:40.240 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46665.log 2024-04-23T21:24:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.58596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58596.log.gz 2024-04-23T21:24:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.50114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50114.log.gz 2024-04-23T21:24:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48604.log 2024-04-23T21:24:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.46665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46665.log.gz 2024-04-23T21:24:40.241 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/ceph-client.admin.48604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48604.log.gz 2024-04-23T21:24:40.259 INFO:teuthology.orchestra.run.smithi111.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-23T21:24:40.473 INFO:teuthology.orchestra.run.smithi111.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-23T21:24:40.846 INFO:teuthology.orchestra.run.smithi111.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-23T21:24:41.316 INFO:teuthology.orchestra.run.smithi111.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-23T21:24:41.410 INFO:teuthology.orchestra.run.smithi192.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-23T21:24:42.003 INFO:teuthology.orchestra.run.smithi111.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-23T21:24:42.780 INFO:teuthology.orchestra.run.smithi192.stderr: 95.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-23T21:24:42.802 INFO:teuthology.orchestra.run.smithi111.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-23T21:24:42.804 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-23T21:24:42.804 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m2.760s 2024-04-23T21:24:42.804 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m7.052s 2024-04-23T21:24:42.804 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.417s 2024-04-23T21:24:43.570 INFO:teuthology.orchestra.run.smithi192.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-23T21:24:45.592 INFO:teuthology.orchestra.run.smithi192.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-23T21:24:45.594 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-04-23T21:24:45.594 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m5.514s 2024-04-23T21:24:45.595 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m12.669s 2024-04-23T21:24:45.595 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.517s 2024-04-23T21:24:45.595 INFO:tasks.ceph:Archiving logs... 2024-04-23T21:24:45.596 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi111/log 2024-04-23T21:24:45.597 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T21:24:46.775 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi192/log 2024-04-23T21:24:46.776 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T21:24:48.502 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-23T21:24:48.512 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-23T21:24:48.512 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T21:24:48.515 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-23T21:24:48.593 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T21:24:48.603 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T21:24:48.603 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T21:24:48.605 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T21:24:48.619 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-23T21:24:48.622 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-04-23T21:24:48.624 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T21:24:48.624 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-23T21:24:48.624 INFO:teuthology.orchestra.run.smithi111.stdout:^- hv01.front.sepia.ceph.com 2 6 377 36 +560us[ +560us] +/- 569ms 2024-04-23T21:24:48.625 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1085us[ -880us] +/- 65ms 2024-04-23T21:24:48.625 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +1123us[+1123us] +/- 62ms 2024-04-23T21:24:48.625 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:24:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T21:24:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-23T21:24:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:^- hv01.front.sepia.ceph.com 2 6 377 37 +586us[ +588us] +/- 569ms 2024-04-23T21:24:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -1087us[-1084us] +/- 65ms 2024-04-23T21:24:48.628 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +1134us[+1136us] +/- 62ms 2024-04-23T21:24:48.629 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T21:24:48.629 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T21:24:48.666 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T21:24:48.666 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T21:24:48.759 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T21:24:48.838 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T21:24:48.912 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T21:24:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:24:48.940 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-23T21:24:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:24:48.969 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T21:24:48.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T21:24:49.025 INFO:teuthology.task.internal:Duration was 638.204637 seconds 2024-04-23T21:24:49.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T21:24:49.058 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T21:24:49.058 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T21:24:49.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T21:24:49.097 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T21:24:49.099 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T21:24:49.564 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T21:24:49.564 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-23T21:24:49.564 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T21:24:49.624 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-04-23T21:24:49.624 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T21:24:49.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T21:24:49.653 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T21:24:49.666 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T21:24:49.719 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T21:24:49.719 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T21:24:49.732 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T21:24:49.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T21:24:49.882 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T21:24:49.882 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T21:24:49.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T21:24:49.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T21:24:49.941 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T21:24:49.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T21:24:49.968 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-23T21:24:49.992 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-04-23T21:24:50.023 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T21:24:50.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:24:50.041 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T21:24:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T21:24:50.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T21:24:50.087 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T21:24:50.088 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi111 2024-04-23T21:24:50.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T21:24:50.126 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-23_20:16:13-rbd-main-distro-default-smithi/7670212/remote/smithi192 2024-04-23T21:24:50.126 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T21:24:50.158 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T21:24:50.158 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T21:24:50.161 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T21:24:50.208 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T21:24:50.219 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T21:24:50.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T21:24:50.253 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T21:24:50.253 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T21:24:50.255 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T21:24:50.267 INFO:teuthology.orchestra.run.smithi111.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 21:24 /home/ubuntu/cephtest 2024-04-23T21:24:50.269 INFO:teuthology.orchestra.run.smithi192.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 21:24 /home/ubuntu/cephtest 2024-04-23T21:24:50.270 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T21:24:50.322 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T21:24:50.433 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit} duration: 638.2046365737915 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-23T21:24:50.433 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T21:24:50.516 INFO:teuthology.run:pass